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system to filter product records to identify a 
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METHOD AND SYSTEM FOR NETWORK-BASED DECISION PROCESSING AND 
FOR MATCHING REQUESTS FOR PROPOSALS TO RESPONSES 

RELATED APPLICATIONS 
5 This application is a Continuation-In-Part application of U.S. Application 

No.09/396,215, filed September 15, 1999, which is incorporated herein by reference in its 
entirety. Also, this application claims the benefit of U.S. Provisional Application No. 
60/201,526, filed May 2, 2000. 

10 FIELD OF THE INVENTION 

The present invention relates to computer methods and systems for a network-based 
evaluation engine. 

BACKGROUND OF THE INVENTION 

1 5 A wide variety of products and services (collectively referred to herein as "products") 

are available over the Internet and featured on-line. To make informed choices, users of the 
Internet need effective on-line tools to assist them in choosing or ranking products. 

There are existing on-line services that assist a user in selecting a product. One such 
existing service is a website "peisonalogic.com." This service obtains two types of data from 

20 a user. First, a user identifies the types of products he/she is looking for so as to eliminate 
from consideration other available, but irrelevant, products. Second, the user provides input 
concerning the importance of several characteristics of the desired product The user is 
presented with a name of each characteristic along with boxes where the user can check 
whether the characteristic is less important, no opinion, or more important. Based on this 

25 input, this existing service outputs a list of products, each product rated on how it meets the 
user's overall preferences. The list is displayed such that the most suitable products are listed 
first 

Existing product-selection services suffer from a variety of limitations. For example, 
they do not allow a user to compare various characteristics of products so as to indicate their 
30 relative importance to each other with respect to the user's preferences. Instead, the user 
simply considers each characteristic separately and weighs its importance in isolation. 
Furthermore, after the suggested choices have been proposed and presented to the user, the 
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user cannot dynamically redefine weights assigned to various characteristics of the products 
so as to reevaluate the selections. Another limitation includes the inability of the existing 
systems to analyze the choices made by other users whose demographics are similar to the 
present user so as to provide shortcuts in the decision process. Also, the existing services do 
5 not provide a substantially-natural language-query capability. Moreover, the existing systems 
are incapable of allowing their service and product providers to respond to user's preferences. 
These and various other drawbacks of the present systems make them less efficient and 
intuitive, and unable to sufficiently accurately capture user's preferences. 

There is stand-alone software that provides a more intuitive approach to the decision- 

10 making process. See "Group Decision Support Software" User Manual, 1998 Expert Choice 
Inc. It, however, lacks the features that would make it adaptable for supporting a wide variety 
of users of an Internet-based service. For example, it does not integrate database filtering, the 
collection and comparison of user's profiles, substantially natural language input, as well as 
various other desired properties. 

1 5 Thus, there is a need to provide an intuitive and flexible Internet- (or another 

network-) based system that would effectively assist a large number of on-line users in 
choosing products and services. 

There is a significant body of work in the areas relating to decision processing that is 
useful for implementing improved decision processing systems. See, e.g., T. L. Saaty, "A 

20 Scaling Method for Priorities in Hierarchical Structures", J. Math Psychology, Vol. 1 5, 

pp. 234-281, 1977; P. T. Harker, and L. G. Vargas "Theory of Ratio Scale Estimation: Saaty's 
Analytic Hierarchy Process", Management Science, 33, pp. 1383-1403; F. Zahedi, "The 
Analytic Hierarchy Process-a Survey of the Method and Its Applications", Interfaces, Vol. 16, 
pp. 96-108 (1986); T. L. Saaty, The Analytic Hierarchy Process, McGraw-Hill, New York 

25 (1 980); E.H. Fonnan, "Decision Support for Executive Decision Makers", Information 
Strategy: The Executives Journal, Vol. 1, Number 4, Summer 1985, Auerbach Publishers, 
Pennsauken NJ; P. T. Harker, "Alternative Modes of Questioning in the Analytic Hierarchy 
Process", Math Modeling, Vol. 9, No. 3-5, pp. 353-360, 1987, Pergamon Journals; Forman, 
E. H., Saaty, T. L., Selly, M* A., & Waldron, R. Expert Choice, Decision Support Software, 

30 McLean, VA, 1 983; M. Gondran, M. Minoux, Graphs and Algorithms, Search for" the 
connected component containing the vertex algorithm, pg 1 5, John Wiley & Sons; T. L. 
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Saaty, Decision Making/or Leaders, 1995/1996 Edition, RWS Publications, Pittsburgh, PA 
(1985); and T. L. Saaty, Fundamentals of Decision Making and Priority Theory, Vol. 6, RWS 
Publications, Pittsburgh, PA (1 994). All of the above publications are incorporated herein by 
reference. 

5 

SUMMARY OF THE INVENTION 
A preferred embodiment of the present invention comprises an Internet portal or 
platform that enables users to evaluate various products described in one or more databases 
accessible through the portal. A stored description of a product is referred to as a "product 

1 0 record." The term "record" in general refers to any storage of relevant information in a 
database or otherwise as known in the art and is not limited to any particular storage 
organization or configuration. Similarly, the term "database" is not limited to a specific 
database or type of database and, as understood by a person skilled in the art, may refer to 
various types of storage such as files managed by an operating system. Preferably, the stored 

15 description of each product includes a set of characteristics of the products. (The 

characteristics may also be referred to as criteria, factors, and objectives, and the products 
may be referred to as alternatives; both products and alternatives also relate to services as 
well as products). Preferably, at least some of the stored characteristics of each product have 
already been rated by an expert in the field applicable to the products at issue. Thus, the 

20 stored data for the characteristics includes an indication of their ratings, which can be explicit 
or can be derived from other data as described subsequently. For example, in the case of a 
car, these characteristics may include price, safety, roominess, features, and performance. For 
these (and other) characteristics, one or more experts have preferably already provided 
ratings-related information that has been stored in a database in connection with each car 

25 model. For example, a high-quality car would have a high rating for the quality characteristic 
and an expensive car would have a low rating for the price characteristic. 

Furthermore, data stored for each product preferably includes subcharacteristics for at 
least some of the characteristics. For example, such subcharacteristics relating to the 
"features" of a car may include power locks, cruise control, CD player, moon roof, etc. In 

30 general, the characteristics and subcharacteristics can be represented as a hierarchy having 
two or more levels. Each collection of lower level subcharacteristics belonging to the same 
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characteristic (at any level of hierarchy) is referred to as a "cluster." Preferably, one or more 
experts assign priorities to the subcharacteristics at the lowest level, and higher-level 
priorities are derived based on user inputs or default inputs selected by the use with his/her 
profile. The ratings may be derived from a utility curve that rates an absolute property of a 

5 product The curve may, for example, provide a relationship between the actual top speed of 
a car and a specific rating (or score) associated with that speed. 

In a preferred embodiment, a user communicating over the Internet graphically enters 
relative importance of product characteristics in achieving a goal. In this discussion, the 
terms objectives (and subobjectives), characteristics (and subcharacteristics), and criteria (and 

10 subcriteria) may refer to both actual features and a user's preferences or judgments, as will be 
understood by a person skilled in the art from a given context. Thus, the terms characteristics 
(and subcharacteristics) and criteria (and subcriteria) may cover the concept of objectives and 
sub-objectives as will be understood by a person skilled in the art from the context. The 
relative importance is preferably entered using pair-wise comparisons that are conducted with 

15 two parallel sliding bars, or a single bar with a slider that moves bi-directionally from the 
fulcrum, for each compared pair of the product's characteristics. The system then processes 
this data to determine the weights of the characteristics. The weights indicate a user's 
preferences; the terms "weights" and "priorities" relate to the same concept and can be used 
interchangeably. The user can also specify relative importance of subcharacteristics by 

20 conducting pairwise comparisons (also known as "trade-offs"). 

In addition, a user preferably provides filtering decisions so as to eliminate from the 
available alternatives clearly unacceptable products. These filtering decisions can also 
eliminate or add certain filtering questions or ^characteristics. If filtering unduly reduces 
available alternatives, the user is notified appropriately and preferably is provided with a 

25 capability to change the filtering choices in response. 

A preferred system analyzes the relative importance of judgements corresponding to 
the characteristics and subcharacteristics entered by the user as pairwise comparisons (or 
otherwise) and assigns specific priorities to subcharacteristics and characteristics by applying 
a decision engine running the analytic hierarchy process (AHP). For a detailed description of 

30 the AHP see Saaty, Decision Making For Leaders, cited above and incorporated herein by 
reference. Based on the determined judgements, the system assigns priorities using the AHP 
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to the objectives indicating how important each objective is to the user's goal. Once 
priorities are calculated and rating scales have normalized the product records, a process 
called "synthesis" is used to assign an overall value score to each alternative so that they can 
be ranked- Thereafter, several top choices in order of priority are displayed to the user. In 
5 addition, the system displays how the characteristics and subcharacteristics contributed to the 
priority of the products by graphically illustrating the weights of the characteristics. The user 
can then flexibly adjust the weights of the characteristics and in response the system 
dynamically re-computes the priorities assigned to the products. This information and 
interface can be provided not only for the top choices, but also for other alternatives selected 
10 by the user. 

Preferably, after the user enters pairwise comparisons, a graphical user interface 
supplying the user with a horizontal histogram listing characteristics or subcharacteristics is 
displayed, along with their current weights, rank order, and graphical slide bars to adjust the 
weights. Also, preferably, an inconsistency ratio indicating inconsistencies in the user's 

15 relative preferences is computed and displayed. Based on this information, the user can 
adjust, validate, and recalculate his/her inputs. 

The preferred system has a capability of searching its database and identifying users 
who have previously participated in the decision-making process and whose demographic 
profiles are similar to the profile of the user presently engaged in the decision making 

20 process. The system runs regressions on the weights previously assigned to various 

characteristics by such users and then computes a sample selection of products based on these 
averaged weights. This technique can be employed to eliminate the need for a user's input 
during the selection process entirely. It can also be employed to eliminate a need to enter 
certain pairwise comparisons, for example, for all or some subcharacteristics. Instead of 

25 entering such comparisons, the user may select a decision profile assigned to the 
subcharacteristics by similar users. 

Another embodiment of the present invention comprises a method of assisting a user 
over a network that enables users to evaluate various products. The products are described in 
one or more dynamically-generated data tables accessible through the network. Such a (lata 

30 table can be dynamically generated by products or services providers in response to the user's 
preferences. This and related embodiments are now discussed. 



5 



WO 01/20530 



PCT/USOO/2550<> 



A preferred embodiment comprises enterprise software designed to function across 
the Internet. One version can be installed as a stand-alone system; a second version can be 
installed by a practitioner (for convenience, denoted often herein by "EC," for 
ExpertCommerce, but referring generally to anyone practicing the invention) of the subject 
5 invention on servers dedicated to the client, leased or owned by EC and managed and 
maintained by EC. The client receives the same functionality as if it were hosting, with the 
exception of the responsibility for maintenance, data storage, and upgrades, which is EC's. 
The latter model is known in the art as an Application Service Provider (ASP) solution. 

As discussed above, a stored description of a product, alternative, or bid is referred to 

10 as a "product record." The term "record" in general refers to any storage of relevant 

information in a database and is not limited to any particular storage format Similarly, the 
term "database" is not limited to a specific type of database and, as understood by a person 
skilled in the art, may refer to various types of storage, such as files managed by an operating 
system. The stored description of each product includes a set of characteristics of the 

15 products. Preferably, an expert has defined rating scales for any raw data in the record. 
Ratings are applied to the data during the application's processes. Thus, the stored data for 
the characteristics includes an indication of their ratings type, which can be explicit or can be 
derived from other data as described subsequently. For example, in the case of an RFX 
(RFX, as is known in the art, is a term used to refer to both RFPs (requests for proposals) and 

20 RFQs (requests for quotations)) for an electronic component, these characteristics or 

objectives may include business terms, product specs, and independent ratings. For these 
(and other) characteristics, one or more experts preferably have already provided 
ratings-related information, that has been stored in a database, in connection with each modeL 
For example, a component with a high number of failures per 1000 would receive a low 

25 rating under its corresponding covering objective, which would lower the final value score for 
the top level objective independent ratings, while the same component may have a very short 
lead-time for delivery, which would translate to a relatively high score under one of the 
covering objectives within the top level objective business terms. 

Furthermore, data stored for each product preferably includes subcharacteristics, 

30 subfactors, sub-objectives, or covering objectives for at least some of the characteristics. For 
example, such subcharacteristics relating to the product specs of a component may include 
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bus speed, power usage, calculations per second, and average running temperature. In 
general, the characteristics and subcharacteristics can be represented as a hierarchy having 
two or more levels and parent-child dependencies. Each collection of lower level 
subcharacteristics belonging to the same characteristic (at any level of hierarchy) is referred to 
5 as a "cluster" In one preferred embodiment, buyers assign priorities to the characteristics at 
any level of the hierarchy, based on what their breadth of knowledge permits them to do. The 
remaining top level or sub level characteristics can be prioritized by selecting a profile that 
has been completed by a content expert, who has completed it with respect to a particular 
usage or demographic grouping. An example of this is a profile for thin client terminals 

10 being made available to a purchaser who has enough knowledge to conduct tradeoffs on the 
top level characteristics of business terms, product specs, and independent ratings, but does 
not feel comfortable conducting tradeoffs between bus speed and running temperature. This 
buyer would load the profile in accordance with the intended use of the component in order to 
have the lower level judgments populated. The buyer may then create a personal profile that 

15 contains the expert's lower level judgments across all subcharacteristics, merged with the 
buyer's organizationally-focused upper level judgments. 

A preferred system has a capability of searching its database and identifying users 
who have previously participated in the decision-making process and whose demographic or 
usage profiles are similar to the profile of the user presently engaged in the decision-making 

20 process. The system runs a regression on the weights previously assigned to various 
characteristics by such users, and then computes a selection of products based on these 
newly-calculated priorities. This method can be employed to eliminate user input during the 
selection process entirely. It can also be employed to eliminate entering certain pairwise 
comparisons, for example, for all or some subcharacteristics. Instead of entering such 

25 comparisons, the user may select profile weights assigned to the subcharacteristics by similar 
users. This feature can also update global profiles (those available to all users) to keep them 
current with the market. 

In a preferred embodiment, a user communicating over the Internet graphically enters 
relative importance of product characteristics in selecting a particular alternative. In this 

30 discussion, the terms objectives (and sub-objectives), characteristics (and subcharacteristics), 
and criteria (and subcriteria) refer to the elements that a buyer is using to evaluate an 
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alternative, as will be understood by a person skilled in the art from a given context. Thus, 
the terms characteristics (and subcharacteristics) and criteria (and subcriteria) may cover the 
concept of objectives and sub-objectives. The relative importance is preferably entered using 
pairwise comparisons ("tradeoffs") that are conducted with two parallel sliding bars, or a 
5 single bar with a slider that moves bi-directionally from the fulcrum, for each compared pair 
of the product's characteristics. The system then processes this data to determine the weights 
of the characteristics. The weights indicate a user's preferences (the terms "weights" and 
"priorities" relate to the same concept and are used interchangeably herein). 

A preferred system analyzes the judgements that correspond to the relative importance 

10 of the characteristics and subcharacteristics entered by the user as pairwise comparisons (or 
otherwise) and assigns specific priorities to such subcharacteristics and characteristics by 
applying a decision engine running an analytic hierarchy process (AHP). Again, see Saaty, 
Decision Making For Leaders, cited above and incorporated herein by reference, for a 
description of AHP, a process well-known to those skilled in the art. Based on the 

1 5 determined weights, the system assigns priorities, using AHP, to the objectives, indicating 
how important each characteristic is to a decision. 

In addition, the application preferably provides filtering questions for the buyer to 
complete, so as to eliminate from the available alternatives clearly unacceptable products. 
These filtering questions can also eliminate or add other filtering questions or 

20 (subcharacteristics to the model. If filtering unduly reduces available alternatives, the user is 
notified appropriately and preferably is provided with a capability to change the filtering 
choices in response. In an RFX evaluation-and-implementation embodiment of the invention, 
filtering questions are not immediately run against an existing database. Instead, they are first 
used to generate questions that are made available to suppliers. The form containing these 

25 questions is made available via the Internet. Suppliers are notified of the RFX, preferably by 
email, but any other method is possible including but not limited to fax, phone call, mail, or 
wireless medium. Once notified, the supplier opens the RFX form via the Internet and 
completes the questions generated by filters, as well as the questions corresponding to lowest 
level subcharacteristics (known as covering objectives). Once this form is complete it 

30 becomes a product record. This record is now validated by running the SQL generated by the 
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filters against the supplier's answers, to verify that the supplier's alternative or bid has met 
the acceptable level defined by the buyer. 

Once the product record is created, ratings are used to normalize the raw data in the 
product record. These value scores may be derived from a utility curve which rates an 
5 absolute property of a product, quantified verbal rating scales, or explicit interval step scales 
for non-continuous absolute data. Any scale, for example, provides a relationship between 
the actual top speed of a component and a specific rating (or score) associated with that 
speed. The result is a number between 0 and 1 that represents a value score with respect to a 
covering objective. 

10 Thereafter, the software application performs synthesis to bring together the value 

scores and the priorities of the objectives. The result is an overall value score for each 
alternative. All alternatives that made it through filters are ranked and displayed to the user. 
In addition, the system displays how the characteristics and subcharacteristics contributed to 
the priority of the products by graphically illustrating the weights of the characteristics. The 

15 user can then flexibly adjust the weights of the characteristics and, in response, the system 
dynamically re-computes the priorities assigned to the products. This information and 
interface can be provided not only for the top choices, but also for other alternatives selected 
by the user. The effect of this dynamic sensitivity testing is a real-time re-ranking of the 
alternatives that allows a buyer to build confidence intervals, quantify sensitivity, play out 

20 scenarios, and justify a decision. All of these analysis or sensitivity tools provide graphical 
and printable outputs. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 illustrates overall architecture of a preferred embodiment. 
25 Figs. 2-4 illustrate overall organization of web pages of a preferred service. 

Fig. 5 illustrates a flowchart of software for selecting a car using a preferred service. 
Fig. 6 illustrates an example of entering pairwise comparisons. 
Fig. 7 illustrates a flowchart of software for administering entering demographic 
questions. 

30 Fig. 8 illustrates a flowchart of software assisting a user in prioritizing the 

alternatives. 
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Fig. 9 illustrates a flowchart of software for administering pairwise comparisons. 

Fig* 10 illustrates a flowchart of software for administering filtering questions. 

Fig. 1 1 illustrates a flowchart of software for determining sample data based on 
demographic information. 
5 Fig. 12 illustrates a flowchart of software for locating a user's record. 

Fig. 13 illustrates a flowchart of overall software architecture controlling user 
interaction with a preferred service. 

Figs. 14-18 illustrate output screens of a preferred embodiment, including sensitivity 

analysis screens. 
1 0 Fig. 1 9 illustrates a utility curve. 

Fig. 20 illustrates interactive selection of characteristics and subcharacteristics for 
sensitivity analysis. 

Fig. 21 A illustrates pairwise comparison of two characteristics. 

Fig. 21B depicts a matrix that includes ratios for pairwise comparisons. 
1 5 Fig. 22 illustrates computation of inconsistency. 

Figs. 23 and 24 illustrate the determination of priorities of products. 

Fig. 25 illustrates a utility curve. 

Fig. 26 illustrates a rating scale. 

Fig. 27 illustrates a feedback process. 
20 Fig. 28 illustrates overall architecture of a second preferred embodiment. 

Fig. 29 illustrates a detailed view of a second preferred embodiment. 

Fig. 30 illustrates a flowchart of software for using a preferred service on a network. 

Figs. 30A and 30B illustrate flowcharts of a buyer using a preferred service to make a 
request, and a seller's response to a request, respectively. 
25 pig. 31 illustrates a flowchart of software of a preferred service of the embodiment of 

Fig. 28. 

Fig. 32 illustrates a flowchart of software for generating Request For Proposals/ 
Request For Quotes of the embodiment of Fig. 28. 

Fig. 33 illustrates a flowchart of software for building preferences of the embodiment 

30 of Fig. 28. 
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Fig. 34 illustrates a flowchart of software for aggregating and manipulating supplier 
responses of the embodiment of Fig. 28. 

Fig. 35 illustrates a flowchart of software for loading data into a decision engine of the 
embodiment of Fig. 28. 

5 Fig. 36 illustrates a flowchart of software for evaluating data with a decision engine of 

the embodiment of Fig. 28. 

Figs. 36A-D show examples of the application of steps 3603 and 3604 shown in Fig. 

36. 

Fig. 37 illustrates a flowchart of software for providing graphical sensitivity outputs 
10 of the embodiment of Fig. 28. 

Fig. 38 illustrates architecture for administering a dynamic tree builder module over a 
network of the embodiment of Fig. 28. 

Fig. 39 illustrates architecture for administering a trade-off module over a network of 
the embodiment of Fig. 28. 
15 Fig. 40 illustrates architecture for administering a utility curve over a network of the 

embodiment of Fig. 28. 

Fig. 41 illustrates architecture for administering a decision engine over a network of 
the embodiment of Fig. 28. 

Figs. 42-44 illustrates architecture for administering of sensitivity graphs over a 
20 network of the embodiment of Fig. 28. 

Figs. 45-55 illustrate samples of input and output screens of the embodiment of Fig. 

28. 

Figs. 46A, 49 A. 50A, 51 A, 54A, and 55A depict screens analogous to those shown in 
Figs. 46, 49, 50, 51, 54, and 55, respectively, but for automobile procurement instead of 
25 aircraft parts procurement. 

Fig. 56 illustrates first and second preferred embodiments. 
Fig. 57 shows the relationships between elements of the first and second preferred 
embodiments and associated figures. 

Fig. 58 illustrates model relationships. 
30 Fig. 59 depicts a model selection page. 

Fig. 60 depicts a profile selection page. 
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Fig. 61 depicts a filters page. 
Figs. 62 and 63 depict tradeoff pages. 
Fig. 64 depicts a fill] results page. 
Figs. 65 and 67 depict summary report pages. 
5 Fig. 66 depicts a transaction report page. 

Figs. 68 and 69 illustrate first and second preferred embodiments, with actor 
relationships. 

Fig. 70 depicts a supplier response form page. 
10 Fig. 71 depicts a head-to-head analysis page. 

Fig. 72 depicts a gap analysis page. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
Fig. 1 illustrates the configuration of the system of a first preferred embodiment. 

15 Users' devices illustrated as 100 communicate over the Internet 120 with the website portal of 
the preferred service. A user's device can be any device capable of communication over the 
Internet (or another applicable network) using, for example, a browser, as known in the art. 
Thus, such a device can be a conventional personal computer, Internet appliance, or 
essentially any other Internet-enabled device. In other embodiments that do not employ the 

20 Internet, such a device should have a capability to communicate over the chosen computer 
network, e.g., an intranet, as known in the art. The portal is symbolically illustrated as 130 
and can be implemented using a server computer, as known in the art, which may range from 
a personal computer to a workstation or a larger computer or a network of computers. The 
choice of a specific computer system for the portal is based on implementation trade-offs as 

25 known in the art. The portal is preferably an Internet-enabled server, but alternatively can be 
another server that supports another network, e.g., an intranet At least one database is stored 
at the server and administered as known in the art. 

Blocks 135-145 broadly illustrate software architecture of the portal server. This 
software includes the decision engine 140 that provides data analysis supporting the decision 

30 making capability and implementing the AHP process. More specifically, the engine 
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identifies the weights allocated by the user to various characteristics and subcharacteristics of 
the products and then prioritizes them in accordance with the user's preferences. 

Block 145 illustrates web pages as known in the art that generally guide and inform 
the user interacting with the portal server and block 135 illustrates application software that is 
5 executed to support the selection processes for various products (services are included in the 
definition of "products")- The decision engine and the application are interfaced preferably 
over the Internet with external sources of data and information. For example, such external 
sources may include databases of various products maintained by third parties, e.g., a 
database of restaurants and their ratings, a database of automobiles and their ratings, or a 

10 database of financial instruments and their ratings, etc. Preferably, the service uses HTML- 
formatted documents to interact with the users, but alternatively can employ other standards. 

Fig. 2 generally illustrates the classes of preferably HTML documents available at the 
portal that provide general information and guidance. In this illustrative embodiment, these 
documents include documents relating to historical information about the company providing 

1 5 the service, see 201 ; documents generally describing the technology employed by the 
preferred system, see 202; preferably chronologically-indexed internal documents that are 
accessible by the users, e.g., press releases, see 203. The information illustrated in Fig. 2 may 
also include chronologically-indexed externally-generated electronic documents such as news 
items regarding the present service. See 204. Electronic documents featuring brief histories 

20 of corporate offices preferably including their vision statements are also stored on the server 
and are accessible to the users. See 205. Also included is an electronic page that has links 
enabling a user to access various departments or individuals of the service who interact with 
customers over the Internet such as human resources, investor relations, potential partners, 
technical support, etc. See 206. 

25 Fig. 3 illustrates exemplary applications that may be provided by the preferred service. 

It should, of course, be noted that the technology discussed herein is not in any way limited to 
specific applications. A person skilled in the art will appreciate that the disclosed technology 
can enable a wide variety of on-line applications and is not limited to a specific use. As 
illustrated in Fig. 3, these applications include a decision tool (i.e., application) for choosing 

30 healthcare providers, see 300; a decision tool for personal investment choices, see 301 ; a 
decision tool for choosing a laptop computer, see 302; a decision tool for choosing an 
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institution of higher learning, see 304; a decision too! for choosing a desktop computer, see 
305; a decision tool for selecting a dining out destination, see 306; a decision tool for 
choosing a camera, see 307; a decision tool for choosing a video recorder, see 308; a decision 
tool for selecting a vacation spot, see 309; and a decision tool for selecting a new car model, 
5 see 310. It is not necessary to describe in detail how each of these applications are 

implemented because based on the exemplary and general application designs, described in 
detail below, various other applications can be built. An exemplary application supporting 
car selection (3 1 0) is described in detail below. 

As illustrated in Fig. 4, the portal includes additional information stored preferably as 

1 0 HTML pages that assists a user in interacting with the decision support applications (decision 
tools). They include electronic documents containing flow charts and descriptions of the 
decision processes used by the service (see 41 0). These documents also include 
alphabetically-indexed electronic documents listing current customers (see 420), as well as 
alphabetically-indexed documents comprising active links to the websites of partners of the 

15 preferred service (see 430). In addition, the server stores demos of how to use various 
applications provided through the preferred service (see 440). 

Fig. 5 illustrates an exemplary application that assists a user in deciding which 
automobile to purchase. At 501 the user invokes this application by making an appropriate 
selection at a web page provided by the server and the system initiates the interaction with the 

20 user. First, at 502, it forwards to the user a page that provides an overview and explains this 
application. Next, at 503, the user is provided with a choice of whether to interact with a 
sample builder subsystem 504 that generates a sample decision based on the decisions of 
similar users or to proceed directly to identifying the user's own preferences. If the user 
decides to build a sample, control is transferred to 504. At 504, the system requests a user to 

25 enter personal information and retrieves from its database the records of users who previously 
made car selection choices and whose personal profiles are similar to the profile of the user at 
issue. To identify such similarly-situated users, the current user enters his/her demographic 
information and the system determines priorities consistently with choices of similar users. 
The user's demographic information may already be stored in the system and retrieved using 

30 identification provided by the user. Based on the profiles of similar users, the system then 
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presents a list of suggested car models to the user. The user may accept such a choice and 
exit the system. 

Otherwise, control proceeds to blocks 505 and 506 where the system, using an 
appropriate database key, retrieves demographic questions and forwards them to the user. 
5 User's responses regarding his/her name, zip code, and gender enables the system to build a 
login key for the user. Alternatively, the user at this point may simply enter his/her login 
name because his/her demographic information has already been stored at the system. Also, 
as understood by a person skilled in the art, such demographic information can be determined 
after entering the application and before block 503. 

1 0 Next, the user is prompted to filter out the alternatives in the database that are clearly 

unacceptable to him/her. At this point, the system retrieves an appropriate database key and 
based on this key the system retrieves the questions that determine the set of cars that are 
relevant to the user (see 507 and 508). These filtering questions may include: car body type, 
year and/or manufacturer that the user wants. Based on user's responses, the system removes 

15 from the consideration all the cars (alternatives) that do not meet the specified requirements. 
The filtering questions are preferably administered as a computerized form where the user 
checks response boxes as known in the art These filtering questions may eliminate future 
filtering questions as irrelevant. Also, based on the answers to the filtering questions, the 
system may determine that certain characteristics should not be considered and certain 

20 characteristics should be added to the set of considered characteristics . 

Next, the system enables the user to identify his/her preferences for the high level 
characteristics of the desired car. These preferences are identified using pairwise 
comparisons that are retrieved from the database using the appropriate key, as illustrated at 
509 and 5 10. In this example the high-level characteristics includes (1) safety, (2) features, 

25 (3) performance, and (4) price. Of course, other characteristics (e.g., maintenance costs) may 
also be specified. The user graphically indicates on a pairwise basis which of these 
characteristics are more important and to what extent By adjusting two parallel sliding bars, 
the user may compare safety to features; features to performance; and performance to price to 
ascertain the relative importance of these characteristics. Other pairs of characteristics may 

30 also be compared. The user interface for the pairwise comparisons is illustrated in Fig. 6. 
The parallel bars at the top portion of the screen are adjusted by the user to indicate relative 
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priorities of the characteristics. The circular chart representation is generated automatically 
as known in the art to reflect relative priorities identified by the position of parallel bars. The 
user can perform various comparisons such as safety to performance, features to price, etc, 
including the exhaustive set of pairwise comparisons. This illustration shows a comparison 
5 of maintenance cost to the cost of an automobile in the embodiment where the maintenance 
cost is one of the high level characteristics Nine dots at the right bottom part of the screen in 
Fig, 6 enable the user to select all such comparisons. However, in this example of four 
characteristics, at the minimum the user must perform at least three comparisons, such that at 
least one characteristic changes in each of these three comparisons. Such necessary 

1 0 comparisons can be chosen by selecting the dots in the diagonal column of three dots. This 
set of necessary comparisons is referred to as a spanning set. A user has the flexibility to 
select the order of entering comparisons. 

After the pairwise comparisons, the user is presented with the option to conduct direct 
weighting to adjust any inconsistencies or errors. In other words, at this point the user can 

1 5 directly enter the weights indicating the relative importance of the characteristics instead of 
providing them implicitly through pairwise comparisons. Thereafter, at 51 1 the user is 
provided with an option to perform additional pairwise comparisons at a lower level for the 
subcharacteristics within each of the characteristics or to allow the system to determine and 
enter default data for the lower level preferences. If the default option is selected, at 512, the 

20 system checks whether the user has a profile stored in the system database that adequately 
describes his/her demographics. If such a profile exists, the user is prompted whether he/she 
wants to modify the profile; if so, the flow is transferred to 5 1 3. Similarly, if there is no 
stored profile with adequate demographic data, the flow is transferred to 5 1 3 where the key 
indicating the storage of relevant questions is retrieved. Based on this key, appropriate 

25 questions (in addition to those administered previously) are retrieved from the database and 
provided to the user. See 514. After the user has responded to the questions, his/her updated 
or new profile is stored in the database. Alternatively, if the profile exists and the user is 
satisfied with its content, the flow is transferred from 512 to 51 5. The complete profile 
includes extended demographic information such as user's profession, age, income level, 

30 educational level, etc. As noted, similar users can be identified based on the profile. 
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At 5 1 5, user's demographic information is stored and his/her profile is matched 
against the database of existing user profiles so as to identify the weights for lower level 
characteristics assigned by the users with similar demographic characteristics. Then, the 
weights for the characteristics assigned by the users most closely resembling the present user 
5 are averaged for each characteristic and stored (see 516). 

Next, the system performs additional filtering based on the questions concerning less 
significant properties of the desired car. The key identifying the storage of these low level 
filtering questions is retrieved at 5 1 7. Based on this key, the system retrieves and forwards to 
the user additional filtering questions. These filtering questions may include questions 

10 covering transmission type, drive axle, and/or brake type, etc. Based on the responses, the 
system performs the second stage of filtering at 519 and displays the number of remaining 
alternatives to the user. If the user determines that there are too few remaining alternatives, 
the user can return to 51 8 and change his/her responses to the filtering questions so as to 
increase the universe of available alternatives. 

15 If at step 5 1 1 the user elected to enter the low level preferences manually, the flow is 

transferred to 520. The system then checks for the profile of the user as discussed above and 
if the profile exists, the user is presented with an option to modify it. If the user does not 
want to modify the profile, control is transferred to 523. Otherwise, as discussed above in 
connection with 513 and 514, the system determines the key for the appropriate demographic 

20 questions and then retrieves such demographic questions from the database, outputs them to 
the user, and receives the user's inpuL See 521 and 522. After user's demographic 
information has been entered and stored as discussed above, control is transferred to 523 
where the user performs pairwise comparisons to identify his/her preferences for 
subcharacteristics. 

25 The comparisons for the subcharacteristics are performed using graphical tool bars as 

discussed above. The lower level subcharacteristics that the user compares at this point are 
subdivided into related sets that correspond to the high level characteristics compared 
previously at step 510. The comparisons at this level are performed only within the related 
sets corresponding to the high level characteristics. For a car, such lower level characteristics 

30 may include the sound system, safety features, interior options, output, and other features. 
For example, at this level the user may compare a CD player to power windows. The user 
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may also decide not to perform certain pairwise comparisons. In this case, the weights 
associated with such subcharacteristics are determined based on the demographic profile of 
the user as discussed above using the stored choices made by similar users. Accordingly, as 
indicated at 525, after each set of pairwise comparisons, the user has a choice of continuing 
5 with low level pairwise comparisons or instructing the system to load the appropriate weights 
determined using the demographic profile as discussed above. As illustrated, if the user 
chooses to perform the next set of comparisons, control returns to 523. The weights 
determined by performing the comparisons are stored. See 526. Also, a user has an option to 
perform verification adjustment of the weights after the pairwise comparisons by entering the 

1 0 subcharacteristic weights explicitly using a graphical tool. 

After all the preferences for the low level criteria have been collected, the system 
identifies a database key relating to the filter administration at the lower level and retrieves 
the questions, administers them to the user, and collects responses. See 528 and 529. This 
filter administration is the same as discussed above for the lower level criteria in connection 

1 5 with 5 1 8 and 5 1 9. Also, as discussed above, if the number of alternatives based on the low 
level of filtering turns out to be insufficient, the user can change his/her filtering responses, 
for example, by not specifying certain choices, so as to increase the number of alternatives. 
Finally, at 530, the system loads data regarding the alternatives remaining under consideration 
after filtering and at 53 1 it loads all the preferences specified by the user from the record 

20 associated with the user. Next, the decision engine, which runs AHP as discussed below, is 
invoked to determine the weights and to apply the weights so as to determine how the 
alternatives match the user's preferences. See 532. Subsequently, at 534, the system saves 
the determined priorities and at 533 the decision engine outputs the selections for the top 
choices using, for example, sensitivity graphs and benefit cost ratios which can be 

25 manipulated by the user to perform "what if scenario analysis as described subsequently. At 
535 the user may choose to modify the weights and recompute the priorities of the 
alternatives or exit the system. 

It should be noted that the products have preferably been rated by experts. In fact, 
each subcharacteristic (preferably at the lowest level of hierarchy) has been rated. Preferably, 

30 the ratings are established by a utility curve as illustrated in Figs. 1 9 and 25. The ratings 
assigned to the products are provided on the Y axis and the properties of the relevant 
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may also decide not to perform certain pairwise comparisons. In this case, the weights 
associated with such subcharacteristics are determined based on the demographic profile of 
the user as discussed above using the stored choices made by similar users. Accordingly, as 
indicated at 525, after each set of pairwise comparisons, the user has a choice of continuing 
5 with low level pairwise comparisons or instructing the system to load the appropriate weights 
determined using the demographic profile as discussed above. As illustrated, if the user 
chooses to perform the next set of comparisons, control returns to 523. The weights 
determined by performing the comparisons are stored. See 526. Also, a user has an option to 
perform verification adjustment of the weights after the pairwise comparisons by entering the 
10 subcharacteristic weights explicitly using a graphical tool. 

After all the preferences for the low level criteria have been collected, the system 
identifies a database key relating to the filter administration at the lower level and retrieves 
the questions, administers them to the user, and collects responses. See 528 and 529. This 
filter administration is the same as discussed above for the lower level criteria in connection 
1 5 with 5 1 8 and 5 1 9. Also, as discussed above, if the number of alternatives based on the low 
level of filtering turns out to be insufficient, the user can change his/her filtering responses, 
for example, by not specifying certain choices, so as to increase the number of alternatives. 
Finally, at 530, the system loads data regarding the alternatives remaining under consideration 
after filtering and at 53 1 it loads all the preferences specified by the user from the record 
associated with the user. Next, the decision engine, which runs AHP as discussed below, is 
invoked to determine the weights and to apply the weights so as to determine how the 
alternatives match the user's preferences. See 532. Subsequently, at 534, the system saves 
the determined priorities and at 533 the decision engine outputs the selections for the top 
choices using, for example, sensitivity graphs and benefit cost ratios which can be 
manipulated by the user to perform "what if scenario analysis as described subsequently. At 
535 the user may choose to modify the weights and recompute the priorities of the 
alternatives or exit the system. 

It should be noted that the products have preferably been rated by experts. In fact, 
each subcharacteristic (preferably at the lowest level of hierarchy) has been rated. Preferably, 
the ratings are established by a utility curve as illustrated in Figs. 19 and 25. The ratings 
assigned to the products are provided on the Y axis and the properties of the relevant 
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characteristic are on the X axis. Such a utility curve enables the system to capture experts 5 
judgements and apply different scores to different properties of the product. For example, a 
top speed of 120 may be 30% more important than a top speed of 80, but a top speed of 160 
may only be only 10% more important than a top speed of 120. Both have a difference of 40 
5 but the return is vastly different. Such a relationship is reflected on the utility curve. 
Different scales and curves are preferably used for different criteria. 

The preferred system utilizes utility curves as a means of developing a unique 
criterion ratings scale. A different specific utility curve that defines ratings is used for each 
rated subcharacteristic. Such a curve is derived by an expert or a group of experts and can be 

10 used dynamically. That is, the database of products may store the absolute value of the car's 
power and its rating would be computed dynamically based on the curve. In this 
implementation to introduce a new product, one does not need to assign all the ratings 
because some of them can be determined dynamically based on the curve. Various experts 
are preferably provided with group tools so as to synthesize their judgments and derive 

1 5 accurate data points on the curve. Preferably, utility curves reflect synthesized judgments of 
multiple decision makers. 

Figs. 14-18 provide samples of output screens. Some of the outputs discussed below 
provide a user with functionality generally referred to as "sensitivity analysis." Sensitivity 
analysis provides a user with a capability to graphically adjust weights of the characteristics 

20 and in response, the system recomputes priorities of the alternatives and displays them in 
prioritized order. Computational principles relating to this functionality are discussed 
subsequently. In Fig. 14 horizontal bars illustrate the weights for product characteristics 
computed, for example, based on the pairwise comparisons. The buttons below show the 
prioritized car models that have been selected based on the user preferences in the order of 

25 priority. In the sensitivity analysis, the user can dynamically adjust the length of the bars so 
as to change the weights of the characteristics and, in response, the system recomputes the 
priorities and displays the alternatives in the new order of priorities based on the changed 
weights. Similarly, in Fig. 1 5 a two-dimensional graph illustrates the relationship of the 
weights and the priorities of the resulting selections, e.g., the cars. This output is referred to 

30 as performance sensitivity graph. The priorities for the cars and their characteristics are 

shown on the vertical axis, and the characteristics are indicated on the horizontal axis. Thus, 
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the weights for each characteristic of each car are shown. The users can dynamically change 
the illustrated vertical bars corresponding to the characteristics so as to change their weights 
and cause the system to recompute the priorities. 

Fig. 16 provides a static comparison of characteristics of two alternatives. This plot is 
5 known as head-to-head comparison. For example, the screen of Fig. 1 6 illustrates a 

comparison between Volvo and Thunderbird, wherein price and maintenance are rated higher 
for Thunderbird and prestige and quality are rated higher for Volvo. Also, the overall score is 
higher for Volvo. This display can be constructed based on the derived weights and ratings of 
the alternatives. 

10 Fig. 17 illustrates a dynamic sensitivity graph where the selected alternatives are 

displayed on the right side and the weights allocated to various characteristics by the user are 
illustrated on the left. As noted, the principles underlying dynamic sensitivity processing are 
described below. A user can dynamically change the weights on the left, and the system will 
recompute the priorities of the alternatives and display them on the right in prioritized order. 

15 Fig. 1 8 provides a plot of the computed value for a characteristic (in this example, 
maintenance) for the selected alternatives (in this example, cars). 

It should be noted that not only can one change the weights of the highest level 
characteristics in the dynamic sensitivity analysis and see the effect on alternative rankings, 
but a user cart choose to see the weights of sub-characteristics and perform sensitivity 

20 analyses on the selected node. As illustrated in Fig. 20, a user can access desired data via a 
single click from a node corresponding to a characteristic or subcharacteristic on a 
hierarchical tree representing the hierarchy of characteristics. 

It should be noted that the user is not limited to generating the displays as discussed 
above and performing sensitivity analysis for the top choices of the products. Instead, the 

25 user is provided with a list of all the products and the user can interactively select any of the 
products and create graphical comparisons as described above. 

Fig. 7 illustrates the process of collecting and storing demographic information. At 
702 the demographic questions are retrieved from the database based on the appropriate 
database key. The database of the system stores different sets of questions indexed by the 

30 corresponding keys. The key supplied by an application determines the set of questions to be 
loaded into the interactive session with the user. At 703, the system determines whether the 
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set of questions corresponding to the supplied key is null. If so, this process terminates. 
Otherwise, flow is transferred to 704 where the questions identified by the supplied key are 
retrieved from the database and provided to the user. 

At 705 the user enters his/her responses to the questions preferably using an electronic 
5 form as- known in the ait and forwards the responses to the server. Then, at 706, the system 
receives these responses and analyzes them, for example, to check that all the required 
responses have been provided and whether the entered values are within acceptable ranges. 
Such validation of data is known in the art. If during this validation procedure an error has 
been detected, control returns to 702. At this point specific deficiencies in the responses may 

10 be identified and the questions are retrieved and presented to the user again. As indicated at 
707, preferably no more than five (or another selected number) such iterations to obtain 
satisfactory responses are permitted. If after five iterations the system still detects an 
unacceptable deficiency in the answers, control returns to a higher level page of the 
application that invoked this process, 

1 5 if demographic information has been properly entered, the system determines at 709 

whether this user's profile has already been stored in the system database. The system 
constructs a database key for the user based on user's name, zip code, and gender. This 
information has preferably been already entered. If this user's profile record has been found 
in the database, control is transferred to 710 where the profile is displayed to the user. At this 

20 point, the user has an option of modifying the existing record or creating a new record. If the 
profile record does not exist or the user indicated that he/she wants to create a new profile 
record, the system creates a new profile record for the user and stores it in the system 
database at a location identified by a unique key constructed as discussed above. See 712 and 
713. If the user decides not to change his/her existing profile, data regarding the present 

25 decision process is added to the existing record. See 711. 

Fig. 8 illustrates the process of prioritizing alternative products based on user's 
preferences. This process begins at 801 . At this point pairwise comparisons have already 
been performed and the weights have been determined and stored in the record associated 
with the user. The weights have been derived from the pairwise comparisons as discussed 

30 subsequently. After the system has loaded from the user's record the weights, at 803, the 
system determines whether the price was one of the decision categories that have been 
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considered. If price was not one of the categories, the flow is transferred to 804. Otherwise, 
the flow is transferred to 805. At 804, the user is presented with a choice, of whether to view 
a graph reflecting how his/her relative preferences could be enhanced with the increased cost. 
In other words, the graph shows additional benefits consistent with the user's choices that 
5 may be available at higher cost. In addition, the user can view such information as a cost- 
benefit table. 

Otherwise, control is transferred to 805 where the system calculates and constructs 
dynamic sensitivity graphs. The sensitivity analysis graphs, discussed above, show the user 
how his/her relative preferences have been combined to prioritize the products and show the 

10 computed weights of the preferences. As discussed above, the user can view the weights 
allocated to the characteristics and can change the weights on the dynamic sensitivity graph 
by interactively dragging the portions of the graph corresponding to specific characteristics. 
This capability permits the user to execute various **what if scenarios. After the user has 
changed the allocation of weights on the graph, the system recalculates the priorities of the 

15 alternatives based on the changed weights and displays the choices in accordance with the 
recomputed priorities. The system can also display to the user dynamic sensitivity graphs for 
the low level subcharacteristics, see 807. The second-level graphs can also be manipulated in 
the same fashion to show how the priorities of the alternatives would be changed if the user 
reallocates the weights. 

20 At 808, the second level graph is constructed. At 809 the user can select to display 

the second level dynamic sensitivity graphs or quit (see 815) or continue displaying other 
graphs. 

If the user chooses to continue with receiving graphical output, control is transferred 
to 8 16 where the user is presented with choices of such graphical displays. The user may 

25 select to build a head-to-head comparison graph, such as illustrated previously in Fig. 16. In 
response, the system generates such a graph and displays it at 817. Flow then proceeds to 820 
where the user can exit the program or display another graphical output. If the user selects to 
display another output, he/she can select to construct the performance graph as illustrated in 
Fig. 15. See 818. In response, the system constructs such a graph and displays it. See 821. 

30 Then, as before, the user may terminate the process or request another display. Another 
available display is a two-dimensional plot as illustrated in Fig. 18. If the user requests this 
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option, the system generates the plot and displays it to the user. See 819 and 822. Then, the 
user can either terminate the process or continue requesting additional displays. 

If at 804 the user chose an option to construct the benefit/cost table, at 8 10 the system 
loads the costs for the prioritized alternatives in the selected subset of the alternatives. The 
5 costs are preferably stored in the database of alternatives (products) of the present system or 
at a third party database accessible to the preferred system. Next, at 81 1, the system 
constructs the benefit/cost table by dividing the normalized benefits of the alternatives by the 
total cost of the alternatives. The normalized benefit is essentially the determined priority of 
the alternative expressed in such a way as to indicate its benefit to the user. These determined 
10 benefit/cost ratios are then stored in connection with the user's record. See 812 and 826. 
Thereafter, the system displays the benefit/cost table and/or graph to the user, see 813 and 
814. The user can thereafter terminate the program or control is transferred to 804 and then 
to 805. 

Fig. 9 illustrates the process of enabling a user to enter the weights of various 

1 5 characteristics of the desired product using pairwise comparisons ("product" includes service, 
as indicated above). It should be noted that although in the preferred embodiment the user 
allocates the weights to the characteristics using pairwise comparisons, after the weights have 
been computed from the pairwise comparisons the user may adjust the weights by entering 
the weights direcdy as numerical data or using graphical tools. 

20 As illustrated in Fig. 9, the comparison process starts at 901 and at 902 the system 

retrieves from its database the characteristics (criteria) to be compared for a particular 
application. For example, as discussed in the example above relating to choosing a car, the 
high level characteristics may include safety features, performance, and price. 

Next, at 903, the user is queried by the system whether he/she wants to perform a 

25 complete set of comparisons or only a minimum required set (referred to as a spanning set of 
pairwise comparisons, or just a spanning set). The full set includes all the pairs of 
characteristics, whereas the spanning set requires only the minimum number of comparisons 
that enable the system to compute the weights allocated to various characteristics. More 
specifically, assuming that there are four characteristics A, B, C, and D that must be assigned 

30 relative weights for the decision process, the full set of comparisons requires comparing A to 
B, C, and D; B to A, C, and D; C to A, B, and D; and D to A, B, and C. The minimum set 
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(i.e., the spanning set) on the other hand requires merely that A be compared to B, B to C, and 
C to D. The spanning set may, of course, include any other chains of comparisons that would 
enable the system to link all four characteristics together. In other words, the comparisons of 
B to D, A to C, and C to D are also acceptable spanning comparisons. 
5 Based on the selection of performing full or spanning comparisons, the system 

displays appropriate graphical sliding bars for the pairwise comparisons as illustrated in the 
example of Fig. 6. It should also be noted that in some embodiments the user may provide 
the comparisons that supplement spanning comparisons, but fewer than the exhaustive set of 
comparisons; for example, the user may provide the comparisons of A to B, B to C, C to D 
10 and A to D. It is preferred that in addition to the spanning set a user enters at least one 
additional comparison so as to enable the system to compute an inconsistency rating of the 
comparisons. 

After the user has entered pairwise comparisons reflecting relative preferences for the 
characteristics using graphical user interface (906), the results are stored as indicated at 907. 

1 5 The system then validates the entered pairwise comparisons. If, at 908, this validation 

process determines that the entered information is insufficient to perform further processing, 
control returns to 902. For example, user's input would be invalid if the required 
comparisons have not been entered. As indicated at 909, the system permits five (or another 
predetermined number) iterations allowing the user to enter valid pairwise data. If after the 

20 predetermined number of iterations the user still failed to enter valid data, control flow 

returns to the higher level application execution where the user, for example, can review an 
example of how to provide the comparisons. Alternatively, at this point the system may 
retrieve and use stored default weights for the characteristics at issue that have not been 
properly specified by the user so as to enable the user to continue the selection process. Such 

25 default weights are stored in the system and then retrieved as known in the art in response to 
user's failure to provide proper pairwise inputs. 

Next, at 910, the system invokes the decision engine that runs AHP and determines 
the weights for various characteristics based on the pairwise comparisons specified by the 
user. As noted, this inconsistency rating can be determined if at least one more than the 

30 spanning set of comparisons have been performed. The system also determines the 
inconsistency rating for the pairwise data provided by the user. An example of an 
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inconsistency is when the user indicates that a characteristic A is twice more important than 
B, B is twice more important than C, and C is twice more important than A. Inconsistencies 
do not preclude the decision engine from prioritizing the alternatives in accordance with 
user's input. Even if a user is slightly inconsistent, he/she may still be accurately expressing 
5 his/her preferences. See Saaty, Decision Making For Leaders cited above and incorporated 
herein by reference. The inconsistency rating may be useful to alert the user that he/she may 
wish to reconsider the choices. The system also compares the inconsistency rating of the user 
with inconsistency ratings of other users stored in the database. For example, a ratio of over 
10% denotes a generally high level of inconsistency that tells the user that he/she may 

1 0 reconsider some of the judgments entered as comparisons. A 1 0% inconsistency denotes that 
the user is 1 0% less consistent than a perfectly consistent set of pairwise comparisons 
reflecting users' judgments and 90% more consistent than a random set of comparisons which 
are completely inconsistent. 

But, if the user believes that the pairwise input accurately represents his/her opinions, 

1 5 even given the inconsistency, the final decision will accurately reflect the input and no further 
adjustments are necessary. Inconsistency is compared to a set level (e.g., 10%) so as to alert 
the user, and in addition, the user is provided with the inconsistency measure computed from 
a pool of other users who are demographic peers of the user. This information allows the user 
to ensure that his/her decision is within the bounds of validity. To determine the 

20 inconsistencies of similar users, the system identifies user records of similar users based on 
the demographic profile of the current user and their stored records containing the 
inconsistency ratings are examined. The ratings of similar users are averaged (or processed 
in another convenient manner) to arrive at representative ratings. 

The system displays the computed weights of the characteristics as well as the 

25 inconsistency rating. See 91 1 and 912. Note that at 91 1 the system explains the weights and 
inconsistency ratings to the user. This can be accomplished by comparing the user's input to 
standard results of similar users determined as discussed above. The user may choose to 
proceed with instructing the decision engine to prioritize the products based on the 
determined weights or he/she may decide to change the comparisons thereby causing the 

30 system to recompute the weights and the inconsistency ratings. 
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If the user rejects the resulting weights, the flow is transferred to 915 where the 
system provides the user with a capability of performing pairwise comparisons again or to 
performing certain selected pairwise comparisons or to assigning the weight to the 
characteristics directly. If the user decides to assign the weights directly, the flow is 
5 transferred to 91 6 where the user is provided with graphical tools for entering weights, such 
as graphical slide bars. As indicated at 917, the user adjusts such a bar chart so as to enter the 
weights directly. At 91 8, the system stores the directly-entered weights in a file and 
continues the processing. 

The user may also decide to repeat the entire process of pairwise comparisons. In this 

1 0 case flow is transferred from 9 1 5 to 902 and the process is repeated as discussed above. 

Also, at 91 5, the user may decide to perform additional pairwise comparisons. In this case, at 
92 1 , the user is provided with a list of all the characteristics that can be compared at this stage 
and at 922 the user performs the chosen additional comparisons. Thereafter, flow is 
transferred to 906 where the process continues as discussed previously. 

15 If, at 9 1 2, the user has indicated that he/she is satisfied with the weights and the 

associated inconsistency rating resulting from his/her selections, control is transferred to 913 
where the computed weights are saved into a record associated with the user. Thereafter, this 
process terminates and control returns to the application that preferably continues by 
processing the weights so as to prioritize the products. 

20 Fig. 10 illustrates the implementation of filtering. As discussed above, based on a key 

associated with a given application, the system extracts from the database filtering questions 
and forwards them to the user (see 1001-1003). After the user has entered the requested 
information, the responses are provided to the system; see 1004. The questions and responses 
are preferably administered using computerized forms and check boxes as known in the art 

25 The system then verifies the entered responses (see 1005) and if the responses are invalid, 
flow is transferred back to 1 002. Responses can be invalid if they are incomplete or unduly 
inconsistent. As indicated at 1006, the system allows for five (or another predetermined 
number) such iterations prompting the user to enter valid filtering decisions. If the user after 
a predetermined number of iterations still failed to provide valid data, the process terminates 

30 and control returns to the higher level of the program execution indicating failure. 
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If the system verified the choices as acceptable, control is transferred to 1007 where 
the system converts the responses to the filters applicable for filtering the database. This data 
is then saved in connection with the specific user. See 1008. Thereafter, the system reads the 
filtering responses and processes them so as to create preferably an SQL statement for 
5 filtering the database. See 1008-1010. Then, at 101 1, the system searches the database and 
determines a subset of data that meets the filtering requirements. Also, subsequent filtering 
questions may be eliminated as irrelevant based on this level of filtering or additional 
questions may be introduced. Also, as a result of filtering product characteristics (at any 
level) may be eliminated from the consideration or conversely introduced into the analysis. 

1 0 The identified subset is then saved in a record identified with the user. See 1012. If the 
system also determines that as a result of filtering the subset of the remaining alternatives 
became too narrow or conversely that filtering did not sufficiently narrow the alternatives, 
additional filtering may be needed (see 1013). In this case, the system sets a flag for each of 
the choices that requires additional information and this loop is performed for the filtering 

15 questions identified by the flag. See 1015 and 1016. Also, the data obtained as a result of 
filtering questions are stored in a record identified with the user. See 1014. 

Fig. 1 1 illustrates a sample process of identifying priorities based on user's 
demographic information. First, the system loads a key to locate essential demographic 
questions stored in the database. See 1 101 and 1 102. In response, the system retrieves the 

20 identified demographic questions, administers them to the user, and collects the user's 

responses. The system then processes the received data and builds preferably an SQL search 
query so as to identify users with similar demographic characteristics. See 1 103-1 106. Using 
this query, the system retrieves data concerning the persons with similar demographic 
properties including the weights that they have previously selected for the characteristics of 

25 the product at issue. See 1 107. These weights are then applied to each characteristic 
separately at 1 1 08 and saved in a temporary file associated with the user (see 1 1 09). 
Alternatively, the weights can be normalized using another formula adopted by the system. 

The temporary file is then formatted as input to the decision engine (see 1 1 1 0) and 
provided to the decision engine where synthesis is performed to prioritize the alternatives 

30 based on the user's priorities computed as discussed above. The output prioritized sample 
choices are then displayed to the user, including the priorities corresponding to each 
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characteristic. See 111 1. Next, at 1 1 12, the system prompts the user whether to accept the 
proposed weights or to construct another sample by enabling the user to modify demographic 
data or priorities. If the user elects to construct another sample, this procedure repeats and, 
therefore, control is transferred back to 1 102. If the user accepts the resulting sample weights 
5 and decisions or decides to enter his/her own choices, this procedure terminates. See 1 1 1 3. 
Fig. 12 illustrates the steps of locating user's demographics information in the 
database. First, the system receives user's identifying information, such as his/her name, zip 
code and gender. See 1201, 1202. Based on the identifying information, the system searches 
its database of existing users for the record consistent with the entered identifying 

10 information. See 1203. If such a record has not been found, this procedure terminates. See 
1207. Otherwise, if there is a match, the system displays to the user his/her existing 
demographic information stored in the record, such as age, profession, income level, etc. 
The system then prompts the user to identify whether he/she is satisfied with the existing 
information or wants to update it. See 1204. If the user is satisfied, control flow is 

1 5 transferred to 1 206 where this information is stored in connection with the user in association 
with the appropriate key. Otherwise, if the user requires an update to the demographic 
information, his/her profile is cleared so as to accept new information consistently with the 
appropriate key. See 1205. 

Fig. 13 illustrates overall software architecture of typical software applications 

20 executed by the system. This architecture is applicable to the applications outlined above, 
including the car selection application discussed above in connection with Fig. 5. First, at 

1 302 the system collects user's demographic information as described above, including user's 
name, zip code, and gender. This data is then used to construct a database key. Based on this 
key, the system identifies if a record for this user has already been stored. Other demographic 

25 information may also be collected at this point As discussed above, if user's record exists, a 
user may modify it or confirm that the stored information is sufficiently accurate. Next, at 

1303 based on the demographic information, the system searches the database of user records 
to identify applicable choices that other users with similar profiles have already made. As 
discussed above, the system retrieves the weights associated with the existing choices of 

30 similar users and preferably averages them. (Another formula (instead of averaging) can also 
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be employed, as understood by a person skilled in the art). The system then computes sample 
decisions based on these weights. 

In addition, the system has a capability to locate and retrieve user profiles and their 
corresponding weights in order to construct typical profiles having certain demographic 
5 properties. Accordingly, the system has a capability of storing decision weights that captures 
user demographic and priorities. These decisions, for example, are based on the priorities as 
discussed above. 

Subsequently, at 1304 the user receives filtering questions for the products being 
selected in the particular application. Based on the user's responses to the filtering questions, 

10 a subset of the stored product information is created. It should also be noted that based on the 
answers to the filter questions, certain subsequent filter questions may not be delivered or 
certain additional ones may be added. The same holds true for the characteristics and 
subcharacteristics that can be ignored or added based on the filtering responses (e.g., in the 
car selection process if the user is only looking at sedans the criteria for towing capacity and 

1 5 bed length will be removed from the model). Dynamic adjustment of subsequent filter 

questions and considered characteristics in response to the filter questions enables the system 
to dynamically customize the decision process. 

Alternatively, instead of filter questions where a user checks electronic boxes on the 
form, a text box may be presented where a user can simply type key words or phrases and the 

20 system parses and interprets the input (This inpxrt is referred to as substantially natural 

language input). Then, the system determines the subset of alternatives or the filter keys and 
characteristic keys based on the textual input. Various known techniques for natural language 
and keyword queries can be employed for such an input; a preferred procedure is explained 
below. 

25 Preferably, questions are delivered to the user in the form of HTML coded statements 

and the responses are entered into what is known in the art as a standard text box form. The 
user is instructed to use standard sentence structure including all punctuation but to limit 
sentences to standard noun, verb, subject format. A sample question could be: "Describe the 
type of car you would like and tell us about some of the most important features." And a 

30 sample response in the form of a sentence or a combination of phrases could be: "Mike fast 
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Japanese cars with 6 cylinders. I don't like sun roofs but I need cruise control. It must be safe 
and not over $30,000." 

In response, the system parses the statement and breaks it down into the following 
components: likes- (wants) Japanese, safety and high performance; needs- (musts) car, 6 
5 cylinder, cruise, and under $30k; and not like- (exclude) sun roof. The system then provides 
feedback to the user to verify their input: "You have told us that you prefer Japanese made 
cars with high safety ratings and high performances. AH of the cars you want to consider 
must have 6 cylinders, cruise control and cost less than $30k and none of the cars can have a 
sun roof." "If this is correct continue." "If there is more you want to tell us go to the filter 
10 interface." 

After this verification of the text input interpretation, the system, in this example, does 
the following: determines a subset of the alternatives that consists of only the alternatives that 
are cars, with 6 cylinders, cruise and which cost less than S30k; and presents the user with 
only safety and performance characteristics to compare. All other characteristics (size, 

1 5 features, etc.) are removed from the evaluation; the system populates filter question data 
based on the user's answers from above input; and removes any characteristics that are not 
relevant based on the user inputs. 

The system displays to the user interpretation of the text input and asks whether to 
proceed. A user is given the choice of viewing the ranking of alternatives based on this input 

20 (in this case the user receives decision outputs) or otherwise the system delivers additional 
filters to further reduce the subset of products and also delivers pairwise comparisons to 
develop criteria weights 

Returning to Fig. 13, after filtering, the user employs pairwise comparisons to enter 
the weight of the product characteristics. As discussed above, the user employs a convenient 

25 graphical representations of sliding bars to input pairwise comparisons. See 1305. 

Subsequently, the user receives another set of filtering questions so as to further reduce the 
set of considered products. See 1306. If, as illustrated at 1307, the second level of filtering 
creates a subset of data which is inadequate, for example, too few choices remain, control 
returns to 1 306 where the user can modify the answers to the filtering questions. The system 

30 then identifies in the database the alternatives that satisfy the filtering requirements-. See 
1308 and 1309. The subset of the database identified as a result of filtering is then saved in 
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association with the user as indicated at 1310 and 1311. Next, at 1312 the user has an option 
to save these results and return to the home page of the preferred service, see 1 3 1 3 and 1314, 
or continued with the process. If the user elects not to continue, the obtained data is saved to 
his/her record which is tagged unfinished. 
5 Next, additional demographic information may be obtained from the user such as 

his/her age, education level, income, family size, and other demographic data, and stored in 
the database. See 1 3 1 5 and 1 3 1 6. This data may already be stored in the user's record, and in 
this case the user can simply confirm that it is correct. Thereafter, another level of pairwise 
comparisons for lower level characteristics is performed. See 1317. As indicated by the loop 

10 at 1318, the user enters a pairwise comparison for each set of these subcharacteristics 
corresponding to high level characteristics. Subsequently, at 1319 a subset of all the 
alternatives that have not been eliminated by filtering is loaded from the database and at 1320 
the preferences determined during the pairwise comparisons are converted into weights using 
the synthesis. Thereafter, at 1321 the decision engine, running AHP, processes the 

15 alternatives based on the weights and saves the priorities of the alternatives in connection 
with the user's record. Then, the output of the decision process is displayed to the user using 
various formats as discussed above. See 1323. Subsequently, the user can view different 
outputs or terminate the process. 

A known optimization program that employs constraints can also be employed with 

20 the output of the decision process so as to develop portfolio selections from ranked 

alternatives. By administering additional filter questions to define constraints and subjecting 
the user to additional pairwise comparisons, weights can be established to extrapolate bounds. 
Once this is done, the data can be fed into the optimization program and a portfolio can be 
produced. 

25 The following discussion describes in more detail how the system operates so as to 

interpret pairwise input, prioritize the products, and deliver an appropriate output including 
the dynamic sensitivity analysis. Fig. 21 A illustrates a pairwise comparison of two 
characteristics (also referred to as objectives). For the pairs of comparisons, the system 
determines a ratio of the lengths of the entered bar as illustrated in Fig. 21 A. The ratios 

30 representing pairwise comparisons are entered into a matrix as illustrated in Fig. 2 IB where 
the i jth elements of the matrix above its main diagonal contain the pairwise comparisons in 
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the form of the ratio of element i to element j . As discussed previously, the user may conduct 
a set of pairwise comparisons for all of the objectives or the user may choose to load the 
results of previous pairwise comparisons to populate the upper right side of the matrix. The 
ratios represented in grid-filled boxes are the results of the pairwise comparisons. Whether 
5 they were completed by the user or loaded from the database is irrelevant The ratios in the 
white boxes are populated by the software and are equal to one because they represent a 
length divided by itself The ratios in the diagonal-patterned boxes are populated by the 
software. They are the reciprocals of their opposite boxes (i.e., X, Y and Y,X). 

Thus, the diagonal elements of the matrix are set to 1 and the elements below the 

10 diagonal are set to the reciprocals of the elements above the diagonal. The normalized 

principle right eigenvector of the matrix is then computed as follows. The matrix is raised to 
powers until it converges, normalizing the column vectors so that they sum to one at each 
stage. As the matrix converges, each of the column vectors becomes the same and each 
element of each column vector converges to a value that represents the priority of the 

] 5 corresponding element being compared. When the difference between each element's value 
in a vector and the value in the preceding iteration is arbitrarily.small (less than a 
predetermined value), the process terminates. Any of the column vectors can be considered 
and only one needs to be computed because all the vectors will be the same when the process 
converges. 

20 It is not necessary to supply all of the N(N-1) elements above the diagonal of the 

matrix in order to compute the principle right eigenvector. As known in the art (see 
M. Gondran, M. Minoux, Graphs and Algorithms, Search For The Connected Component 
Containing The Vertex Algorithm, p. 15, John Wiley & Sons, incorporated herein by 
reference), the eigenvector can be determined if the elements were produced by a spanning set 

25 of pairwise comparisons, which, as discussed above, connects every characteristic in a given 
set of characteristics with every other characteristic. (The judgment set is the same as the 
spanning set described before). For instance, comparing A to B to C to D forms a spanning 
set, which is sufficient to create a direct or indirect relationship between all characteristics. 
Also, an algorithm discussed in Harker, "Ratio Scale Estimation: Saaty's Analytic Hierarchy 

30 Process, " cited above and incorporated herein by reference, can be used to compute the 
principle eigenvector, whereby elements in the matrix corresponding to missing data have a 
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value of 0, and the diagonal elements are set to 1 greater than the number of missing elements 
in each row (or column). Missing elements in the matrices represent the ratios of the pairwise 
comparisons that could be performed outside of the spanning set For example, comparing A 
to C, B to D, A to D, etc. 
5 As noted, the system also provides a measure of inconsistency so as to indicate how 

inconsistent the sets of pairwise comparisons are, relative to random comparisons, provided 
that a user conducts at least one more comparison than the required spanning set. The more 
comparisons are performed, the more accurate the inconsistency measure becomes because 
the user supplies additional information that may help to identify further inconsistencies. In 
1 0 other words, a user may be inconsistent on the comparison between A and D, but the system 
would not know that unless this comparison has been done. Once all of the pairwise 
comparisons have been computed and represented as priorities, AHP checks consistency of 
the comparisons. A high inconsistency does not invalidate a set of comparisons, it simply 
shows a break from conventional logic. The measure of consistency is displayed as a 
normalized number between 0 and 1 corresponding to the percentage as mentioned above. 
The closer this number is to 0, the matrix is more consistent and when the number is closer to 
1, the matrix is more inconsistent. Thus, higher inconsistency is indicated by higher values. 
As noted, the matrix represents a set of pairwise comparisons of the characteristics 
(objectives) or the results of previous pairwise comparisons. 

The consistency index (C.I.) is defined as: - n) / (n-1). ?w = SUM (PV, X J I N. 
PV denotes the priority vector and V* and "N" refer to the number of characteristics being 
compared ( tc N" refers to total number, M n" refers to instances). After computing as 
illustrated in Fig. 22, the consistency of the matrix is determined and the result indicates the 
consistency of the comparisons. Usually, the acceptable inconsistency is 10%. The random 
consistency index (RX) is a reciprocal matrix with the following average consistencies for 
different N random matrices: 
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30 See, Forman, Ernest H., "Random Indices for Incomplete Pairwise Comparison Matrices" 
European Journal of Operations Research, vol. 48, #1, 1990, pp. 153-155. The consistency 
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ratio (C.R.) is defined as: C.I. / R.L = C.R. The C.R. is what is displayed to the user to let 
him/her know the inconsistency of the comparisons. If the comparisons are inconsistent, the 
user may choose to repeat the comparisons or may accept the inconsistency. 

In discussing the prioritization of products based on priorities and ratings of the 
5 characteristics and subcharacteristics, we consider the hierarchy of characteristics and 
subcharacteristics (objectives and subobjectives). The priorities of the products are 
determined as illustrated in connection with Figs. 23 and 24. Figs. 25 and 26 illustrate utility 
curve and rating scale, respectively. The principle of hierarchic composition or synthesis (see 
Saaty, Decision Making For Leaders cited above and incorporated herein by reference) 

1 0 provides for multiplying the local weights of the subcharacteristics in a cluster by the "global" 
weight of the parent characteristics, thus producing global weights throughout the hierarchy 
of characteristics and subcharacteristics (see Fig. 24). Pairwise comparisons are conducted 
for all sets of characteristics and subcharacteristics selected by the user and matrices are then 
constructed. The decision engine derives product ratings by the ratings of childless 

1 5 characteristics or subcharacteristics determined, for example, using a utility curve described 
above. 

Each column of the matrix represents a childless characteristic or subcharacteristic 
and each row represents an alternative using a spreadsheet format. This is called the 
synthesis matrix. See Fig. 24. The data points representing lowest level subcharacteristics 

20 that have data (i.e., cells in Fig. 24) are derived by locating utility scores (ratings) on the 
utility curves defined for each lowest level characteristic/subcharacteristic in a hierarchy. 
These curves, defined for each lowest level (sub)characteristic, are the absolute measurements 
of the performance of each characteristic of the alternatives. The scores converted using the 
curves are illustrated in Fig. 23. The value assigned to each rating is determined by applying 

25 a utility curve to the stored score. Such a utility curve is illustrated as Fig. 25. A utility 

curve, for example, provides that a score of 200 horsepower may only be slightly better than a 
score of 1 80 horsepower, but 1 80 horsepower can be significantly better than 1 60 
horsepower. These curves or rating scales (see Fig. 26) are developed by experts and do not 
change at least for a duration of time. Each utility rating is multiplied by the weight of its 

30 corresponding subcharacteristics and then added across the columns to derive a total • 
composite rating for an alternative. This rating represents the overall performance of the 
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alternative in terms of its value to the user. The alternative with the greatest composite rating 
is the best alternative. As noted, a typical utility curve and a standard flat ratings scale are 
illustrated in Figs. 25 and 26. 

By providing pairwise comparisons at each level of the hierarchy, the system derives 
5 "local" weights of the characteristics and subcharacteristics. The principle of hierarchical 
compositions or synthesis described in Saaty, Decision Making For Leaders (cited above and 
incorporated herein by reference) is applied to multiply the weights of the subcharacteristics 
by the weights of their parents to derive the "global" weights. The ratings of the 
characteristics or subcharacteristics which have been normalized to be between 0 and 1 

10 (where 1 is the highest possible score) for each lowest level subcharacteristic, are multiplied 
by the "global" weights of the characteristics and then added for all lowest level 
(sub)characteristics. 

In the distributive synthesis, when weights are distributed in the hierarchy of 
characteristics and subcharacteristics, the global priority of the goal (standardized to 1.0) is 

15 distributed to the characteristics, and then to the lowest level subcharacteristics. The goal is 
the purpose of the decision making process, i.e., selecting a car. In one embodiment (referred 
to as closed system or distributive synthesis), the priorities of the lowest level 
subcharacteristics are distributed to the alternatives in the same fashion. Their weights are 
multiplied by the global weights and then all of these values are added to obtain the overall 

20 priority of the alternative. 

In the ideal synthesis, instead of distributing each (subjcharacteristic's priority 
(weight) to the alternatives, the priority is allocated to the alternatives so that the most 
preferred alternative under each (sub)characteristic receives the full priority of the 
(subcharacteristics. Each of the other alternatives receives a priority proportional to its 

25 preference relative to the most preferred alternative. The rationale for this approach is that an 
"ideal" alternative (an alternative having the highest possible rating) serves as a reference and 
receive a total priority (before normalization) of 1 .0 while each of the alternatives have 
priorities proportionately lower. Thus, 1 .0 represents a "standard," so that each real 
alternative receives some fraction of the "standard" depending on how well the alternative 

30 compares to the ideal regarding each characteristic. Although no alternative can receive a 
weight from a (sub)characteristic greater than the (sub)characteristic's weight, the sum of the 
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ideal alternatives 1 weights for a (sub)characteristic is not limited as in the closed system. If 
this were a closed system, then all alternative weights could only sum to 1 because each 
alternative weight is a proportion of that 1 . In an open system, each alternative can have a 
weight of up to 1 because it is not being compared to a closed set of alternatives. It is being 
5 rated against a scale so that it is equal to 1 * the decimal of every weighted factor (the sum of 

which is equal to I). 

If a new, irrelevant alternative is added (or an existing irrelevant alternative is 
removed), the priorities allocated to the existing alternatives under each characteristic do not 
change because the alternative being added (removed) is, by definition, not better than the 

10 ideal for any characteristic. Therefore, the ideal continues to receive the full priority (weight) 
of the respective characteristic. Furthermore, the weights allocated to the other alternatives, 
being proportional to the ideal, would not change either. However, the alternative being 
added (removed) would receive (relinquish) a priority in proportion to its weight against the 
ideal alternative. This means that in the distributive mode (closed system), alternatives give 

1 5 away weight to a new alternative that is added to a set because all of them have to sum to 1 . 
This weight could come from any alternative and can differ from characteristic to 
characteristic. This can cause all ranks to change. In the "ideal" approach, one alternative 
gets the highest score so that when the subset of alternatives is normalized, the ideal 
alternative permits a new alternative to be added without all other alternatives giving away 

20 weight to it, so that none in the original set change rank. After the alternatives receive a 
priority from each of the lowest level (sub)characteristics, a subsequent normalization is 
performed so that the sum of the alternative priorities is 1 .0. 

The sensitivity analysis plots illustrated above are used to investigate the changes in 
the priorities of the alternatives when the weights of characteristics at a given level of the 

25 hierarchy are changed. As noted, "characteristics" or "criteria" may also be referred to as 
"objectives" and therefore in this discussion "objectives" are synonymous with 
"characteristics." Also, instead of discussing characteristics and subcharacteristics, we refer 
here to a hierarchy, as known in the art, of objectives. Thus, objectives form clusters that 
belong to a hierarchy and have a parent objective. As discussed previously, this is the same 

30 as characteristics and related subcharacteristics in a hierarchy. In the general case it is 
possible to have several levels of such clusters of objectives. 
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As noted, the sensitivity analysis is performed to investigate the changes in alternative 
priorities as the weights of the objectives change. It is possible to vary weights of objectives 
anywhere in the hierarchy, in which case a user can look at how the priorities of the 
alternatives vary with respect to either the parent of the cluster for which priorities (weights) 
5 are being changed, or with respect to the goal of the comparison. (The goal is the purpose of 
the decision making process, for example, in selecting a car application, the goal is to find the 
car). 

One approach to sensitivity analysis is known as gradient sensitivity. Consider the 
priorities of the alternatives with respect to the goal, given the weights of the objectives, 

1 0 derived from the pairwise comparisons (or otherwise), as a vector (call this vector 0), the i* 
component of which represents the priority of alternative i. A separate gradient sensitivity 
plot consisting of one line for each alternative 0-1,2, ... n) can be produced for each 
objective (j = 1 , 2, . . . m) in the cluster of objectives being considered. 

Consider increasing the weight of one of the objectives, call it j, in the cluster below 

1 5 the goal, and simultaneously decreasing the weight of the other objectives in proportion, such 
that (1) the ratios of the weights of the other objectives to one another remains the same, and 
(2) the total weight equals 1. In the limiting case when the weight of objective j is increased 
to 1 , the weights of the other objectives will become zero. The priorities of the alternatives in 
this limiting case can be computed as the priorities of the alternatives relative to the j* 

20 weight, which is now 1 . These weights can also be put in a vector (call this vector j). Since 
the synthesized weights for the objectives G) and the alternatives (i) are linearly related, these 
two vectors (i and j) can be used to plot straight lines, one line for each alternative, on axes 
where the y axis represents the alternatives' priorities and the x axis represents the weights of 
the objective being varied (objective j). For example, two defining points for the line 

25 corresponding to alternative i are (1 ) the point representing the priority of the alternative i 
given the original weights of the objectives (the y value from the i* element of vector 0 
described above, and the x value, the weight of the j* objective, and (2) the point representing 
the priority of the i* alternative relative to the j* objective (the y value from the i* element of 
the j* priority vector, and the x value equal to 1). The lines are extended to the x axis where 

30 the height of each line represents the alternatives' priorities when the j* objective's weight is 
decreased to 0. A gradient sensitivity plot consisting of n alternative lines is produced for 
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each of the m objectives in the cluster of objectives being considered. In this case, the cluster 
is usually formed \vith top level characteristics (objectives). But, it can also include both 
characteristics and subcharacteristics. 

The sensitivity plots are a function of the vectors available from the ideal or 
5 distributive synthesis processes described above as understood by a person skilled in the art. 
They differ in how much information is displayed and the way the information is displayed. 
Dynamic and performance sensitivity plots (see, e.g., Figs. 14 and 15) are constructed using 
the same linear relationships as the Gradient sensitivity, with different graphical 
representations. The two-dimensional performance-sensitivity plot is a subset of the full 
1 0 performance sensitivity plot, plotting alternative priorities as a function of any two of the m 
objectives at a time. A difference (head-to-head) sensitivity plot depicts the priorities for 
each of two alternatives with respect to each of the m objectives. 

By applying a process called feedback to an existing decision regarding the preferred 
products, instead of showing how alternatives performed with respect to the characteristics 
1 5 and subcharacteristics, the system can show how important characteristics or 

subcharacteristics are with respect to selected alternatives. Thus, a user can use the 
alternatives to study which characteristics or subcharacteristics are usually important to the 
decision and then streamline the selection process. For example, if a user previously thought 
that price was very important in his/her decision making model, to test the validity of this 
20 view, he/she may consider top alternatives and apply the feedback process with respect to the 
top-level characteristics. For example, if alt top alternatives have similar price, then price 
may become less important in the selection. The feedback process enables the user to 
compare any level of characteristics with respect to alternatives and to derive new weights for 
the top level characteristics. These new weights are then aggregated with the original weights 
25 of the top level characteristics in a supermatrix as discussed below. The result is a more 
accurate decision due to the use of more informed weights for the characteristics. 

To implement this process, the resultant matrices reflecting the comparisons as 
discussed above are transformed into a supermatrix which contains the ratios for all 
characteristics and alternatives. (The process of creating a supermatrix is known in the art 
30 and explained in Saaty, Decision Making With Dependence: The Analytic Network Process', 
RWS Publications, Pittsburgh, PA 1996, incorporated herein by reference). Priority vectors 



38 



WO 01/20530 



PCT/USOO/25506 



are calculated, as known in the art, and the sensitivity of characteristics with respect to 
alternatives can be displayed in the fonn of a sensitivity plot. 

Fig. 27 illustrates a feedback process that can be employed by this system. As 
illustrated at 2701, the decision process outputs prioritized alternatives to the user. Next, at 
5 2702, the user may consider specific characteristics (objectives), at high or low levels, in 
connection with a given alternative. If a user decides to do so, the pr&cessing continues; 
otherwise, it terminates. If the processing continues, the system enables the user to choose a 
specific alternative and then the system displays the objectives compared with respect to their 
importance for the alternative. See 2703 and 2704. Thereafter, the user enters new relative 

1 0 importance for the objectives and provides the resultant input to the feedback process. See, 
e.g., Decision Making With Dependents: The Analytic Network Process referenced above and 
incorporated herein by reference. The pairwise comparisons performed on the objectives are 
used to derive priorities for them which are then fed into a supermatrix in the cells 
corresponding to the alternatives and the objectives. The matrix is then processed as 

15 discussed in the above publication. See 2705. As a result the system outputs the decision 
data to the decision engine that subsequently provides the decision output to the user with a 
new prioritization of alternatives. See 2706 and 2707. Then, the user may again choose an 
alternative and reevaluate the objectives. 

Figs. 28-44 illustrate the system of a second preferred embodiment Fig. 28 generally 

20 depicts a plurality of buyer devices 280 1 and a plurality of seller devices 2802 which can 
communicate over the Internet 2803 with a net market 2804 (also referred to herein as a net 
market maker, or NMM, discussed in more detail below) (e.g., an Internet portal service). 
The net market 2804 can be an aggregation site, vertical market, single product market, multi- 
product market, single supplier market, buyer hosted market or any other platform which 

25 aggregates and facilitates commerce. The commerce can be conducted in the form of a 
traditional auction, reverse auction, Dutch auction, dynamic exchange or virtually any other 
form. The buyer or seller device can be any device capable of communication over the 
Internet (or another applicable network) using, for example, a browser, as known in the art 
Thus, such a device can be a conventional personal computer, Internet appliance, Internet- 

30 enabled wireless phone or essentially any other Internet-enabled device. In other 

embodiments that do not employ the Internet, such a device should have a capability to 
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communicate over the chosen computer network, e.g., an intranet, as known in the art. The 
net market 2804 can be implemented using a server computer, as known in the art, which may 
range from a personal computer to a workstation or a larger computer or a network of 
computers. The choice of a specific computer system for the net market 2804 is based on 
5 implementation trade-offs as known in the art. The net market 2804 preferably utilizes an 
Internet-enabled server, but alternatively can use another server that supports another 
network, e.g., an intranet. Also preferably, at least one database is stored at the server and 
administered as known in the art. Preferably, a plurality of databases are in data 
communication with the net market for storing and retrieving information related to user 
10 preferences such as hierarchies, product characteristics, transactional information, aggregated 
data or the like. 

The net market 2804 has a software architecture that includes a network-based 
evaluation module 2805 for dynamically generating requests for proposals ("RFPs") or 
requests for quotes ("RFQs") and matching RFPs/RFQs to responses. The evaluation module 

1 5 2805 may also be referred to herein as a decision guide, a decision engine, a purchase 
evaluation engine, or a dynamic matching engine. This module provides data analysis 
supporting decision making capability and includes the AHP process, as described previously. 
More specifically, the evaluation module identifies the weights allocated by a user, such as a 
buyer, to various characteristics and subcharacteristics of desired products and then prioritizes 

20 them in accordance with the user's preferences and then provides responses including product 
information which preferably match those preferences. 

Referring now to Fig. 29, a general illustration of the software architecture is shown 
with the evaluation module 2805 in communication with the net market 2804 and the net 
market 2804 in communication with the Internet 2803. This view delineates the actions 

25 preferably performed by the net market 2804 and those actions preferably performed by the 
evaluation module 2805. For example, the net market 2804 preferably performs the 
validation of supplier eligibility, notifies and delivers the RFP/RFQ to the supplier, collects 
third party data, processes, validates and aggregates supplier responses and third party data, 
and notifies the successful supplier. The evaluation module preferably generates the 

30 RFP/RFQ, builds preferences using trade-offs or pairwise comparisons, creates a profile 
using a decision engine and compares the profile to data delivered by the net market using 
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AHP, provides sensitivity graphs for interactively selecting and ranking responses to the 
RF P/RFQ, and outputs a value based ranking of the responses. 

Fig. 30 illustrates a part block diagram and a part flow diagram of a preferred method 
of matching RF Qs or RFPs to responses. First, at 3001 , an RFP/RFQ form is loaded and 
5 generated. Preferably, a user, such as the buyer, responds to that module so that it can be 
determined what the user cares to buy and/or evaluate. Next, at 3002, a graphically adjustable 
bar for performing pairwise comparison, as previously described, is preferably delivered to 
the user and the user may adjust the bar according to the user's preference to create a profile 
and a RFP/RFQ. At 3003, the user's preferences or profile are sent out to the net market 

1 0 2804 and the net market searches for eligible sellers or suppliers and eligible suppliers are 
validated. Next, at 3004 the net market 2804 notifies the eligible supplier of the user's 
RFP/RFQ and the supplier can respond to the RFP/RFQ and the net market 2804 collects 
responses and/or data from the suppliers relevant to the desired product or RFP/RFQ to create 
a dynamically generated data table. At 3005, the net market may supplement the data table 

1 5 from other sources, such as third parties or other existing databases, such as product catalogs 
or the like to obtain a more extensive pool of data. Next, at 3006 the net market may verify 
that responses have been made to all the required fieldsand that a sufficient amount of data 
has been collected. Next, the data table is forwarded to the evaluation module 2805 where it 
is aggregated and manipulated at 3007 and preferably formatted to be loaded into a decision 

20 engine running AHP at 3008. Preferably the formatting can be done using SQL scripts, 

however any other suitable technique may be used. Preferably, different scripts are used for 
each distinct implementation to properly format the data. Next at 3009, the decision engine 
running AHP applies a utility curve or rating scale, as previously described, to the data 
supplied by the net market to transform the supplied data. The profile created at 3002 is 

25 opened and the preferences weights are calculated using AHP. The transformed data is 
compared with the weighted preferences to prioritize the alternatives based on the averaged 
weights computed, as discussed above, to produce value-based rankings. The output 
prioritized sample choices or rankings are then displayed to the user, including the averaged 
weights for the characteristics. At 3010(a), the output is preferably captured and savedfor 

30 later analysis, such as aggregate analysis based on user demographics. At 301 0(b), different 
sensitivity tools, as previously described, are delivered to the user for analyzing the data. At 
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301 1, the values or rankings of the bids or responses are transmitted to the net market to be 
forwarded to the user and at 3012, the net market notifies the user or buyer of the bids and 
ranking and the user can interact with the tools or sensitivity outputs. The final step, at 3013, 
is the selection of a response that matches the RFP/RFQ or the buy transaction- A person 
5 skilled in the art will appreciate that steps performed by the net market, as described above, 
may also be performed by the evaluation module. 

Figs. 30A and 30B illustrate a general overview of flow charts of the process, starting 
with a buyer making a request in Fig. 30A and a seller's response to that request in Fig. 30B. 
In Fig. 30A a buyer makes a request by first logging on to a net market at 3020 and chooses 

10 whether or not to use the evaluation module dynamic matching engine. If the buyer uses the 
evaluation module dynamic matching engine, the buyer enters information into the 
application server 3022 required by the net market as well as information required to perform 
the AHP process and create a preference profile to generate an RFP/RFQ. The preference 
profile is then sent to the net market database 3024 and to an evaluation engine dynamic 

1 5 matching engine database 3023 and the net market informs the seller of the RFP/RFQ. If the 
buyer chooses not to use the evaluation module dynamic matching engine, all of the buyer's 
information is forwarded to the net market database 3024 and an RFP/RFQ is forwarded to a 
seller. 

Fig. 30B shows generally a flow chart of a seller's response to a buyer's RFP/RFQ. 

20 First a seller responds by logging into a net market at 3020 and supplies the required 

information to the application server 3022 for the net market and for the evaluation module 
dynamic matching engine. Information needed by the evaluation module dynamic matching 
engine is transferred to the evaluation module dynamic matching engine database 3023 and a 
ranked list of alternatives is returned to the net market application servers 3022 and the buyer 

25 may be informed that all responses have been entered or that bidding is closed. Thereafter the 
buyer may view the ranked list of alternatives on the net market web server 3021 and perform 
dynamic sensitivity analysis to reach a final decision or match the RFP/RFQ with the 
responses. The net market is then informed of the buyer decision and the net market can 
notify the seller of the buyer's decision. 

30 Fig. 31 illustrates the steps which are preferably performed by the evaluation module 

in more detail. At 3103, a hierarchy database is used to store information relating to the 
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criteria node tree. At 3 104, a profile database stores the user created profiles. At 3 1 05, a 
transactional database is used to store data from the ongoing or current or live transaction. 
For example, as the user dynamically alters the preferences using the sensitivity tools, the 
alternatives and their rankings are dynamically altered, and the transactional database stores 
5 this dynamic information as the transaction is ongoing. 

Fig. 32 illustrates the RFP/RFQ generator process of Fig. 31 . First, at 3201 a user 
logs on to the evaluation module using a unique log-on signature as known in the art. The 
log-on signature can be assigned via membership in the net market 2804, assigned from other 
users, such as suppliers, established specifically for a single application, harvested from any 
10 third-party source, or created by the user. Capture of the log-on signature may done via direct 
input into an HTML screen hosted by the net market 2804, read from a file saved locally on 
the user's system, or can be captured using any other means common in the art. 

Next, at 3202, the user's log-on signature is validated by preferably matching the 
signature with a stored list of valid strings. If there is a successful match, preferably access is 
1 5 allowed to a profile of the user's constraints, requirements, and preferences which are 

preferably previously stored. A failure to match the log-on signature will prompt a routine to 
build a profile of the user's constraints, requirements, and preferences. Next, at 3203, 
information that is stored and associated with the log-on, such as a list of all the goods and 
services offered by the net market and qualified to the user will be presented to the user to 
20 facilitate the selection of a good or service to be evaluated or acquired. At 3204, a form 
having questions may be delivered to the user for the purpose of collecting the user's 
requirements and constraints which can be used for filtering, as described above. The 
responses to the questions can be in plain language and the user's requirements or constraints 
can be parsed out of those plain language responses. Also, the form may be static or 
25 dynamically mapped to a series of requests for information which will populate a table, such 
as a hierarchy, for evaluation. The requests can be either subjective or objective. 

The hierarchy mapped to the form is preferably sufficiently inclusive of criteria to 
encompass all criteria necessary to evaluate the desired product or that the user deems as 
substantial. The selection of criteria translates to maintaining a node active rather than 
30 adding nodes to the hierarchical tree. Those criteria not selected by the user cause the 
corresponding nodes to be inactivated. Preferably, the hierarchy allows for dynamic - 
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hierarchical tree building so that users may select from a pre-defined set of criteria to build a 
unique and valid hierarchy. In this way, a user can select from a pool of data points to create 
objectives which act as categories in which data points can be grouped and the data is mapped 
to formulas or ratings scales which transform the data into a comparable format. Also, the 

5 questions and responses to the filtering and criteria questions can be dynamically linked to 
subsequent questions that are dependent upon the prior responses. Next, at 3205, responses 
to the form are verified to assure that required responses have been completed, and if required 
responses are not present, the form may be returned to the user for completion. At 3206, the 
validated form is parsed and the requirements and constraints are preferably separated from 

1 0 criteria which are entered into the decision engine. 

Next at 3208-3212, a hierarchy may be built by applying pre-existing maps 
corresponding to the criteria. This hierarchy is preferably then read into a table to construct 
data-grids and deliver pairwise comparisons or trade-offs and to be analyzed by the decision 
engine. At 3208, lower level subjective and objective questions may also be included to 

1 5 complete the hierarchy. If so, they are also preferably mapped to the corresponding criteria. 
At 32 10, using the completed hierarchy a data grid may be constructed for delivery to 
suppliers or buyers respectively, through an intermediary such as the net market for 
distribution or notification to users. Also, at 32 12, the hierarchy and data-grid may be stored 
in a database. 

20 Referring to Fig. 33, a preferred method of building preferences for a hierarchy is 

shown. The preferences are preferably built using trade-offs or pairwise comparisons, as 
previously described. First, at 3301, a preference table is loaded from the product hierarchy 
database 3103, and at 3302 the database 3103 is queried based on previous log-in 
information; preferably saved, stored, or pre-loaded preference profiles are available. At 

25 3303, a specified default preference may be generated based on business rules associated with 
the user's log-in. At 3304, the user is preferably able to choose to accept the default profile 
or load a preference profile from the previously queried profiles. Also, the user can choose to 
regenerate or make another profile. At 3305, based upon the user response the default profile 
is replaced with a user-selected profile, or the default profile can be left active. At 3306, the 

30 user can adjust the preferences using pairwise comparisons or the trade-off module (shown in 
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Fig. 38). After the desired preference trade-offs have been determined, at 3308 the user may 
store the results as a new profile and the results may be loaded into the profile database 3309. 

Referring to Fig. 34, a preferred method to aggregate and manipulate supplier 
responses is shown. First, at 3401 data is provided to the net market 2804 or market 
5 aggregator, however the present invention is not limited by the data source. The data 
preferably consists of product information, such as XML spec sheets, supplier housed 
catalogs, net market housed catalogs, supplier completed forms, third party housed catalogue 
data, screen scrapings of any web site, completed forms from third party sources, buyer 
completed subjective ratings, or data supplied from any other source. At 3402, the data is 

10 read into a working database for manipulation and aggregation. At 3403, filters, such as 
scripts or any other agents, are applied to the data to order and format the data into a 
predetermined layout. 

Next, at 3404 maps and/or conversion algorithms are applied to the data to transform 
non-numeric data fields into real numbers and at 3405 the data is formatted, for example by 

1 5 applying formulas to the data, for input into the decision engine running AHP. The formatted 
input file is preferably saved into the transactional database. 

Fig. 35 depicts a preferred method of loading the data-grid into the evaluation engine. 
First, at 3501 the formatted input file is opened. At 3502, the questions designated to be 
filtered by the user are parsed out and a table is created containing the parsed-out filter 

20 questions at 3503. An empty data-grid is preferably created at 3504 from the remaining, non- 
parsed out information in the input file and at 3505 the data-grid and filter file are preferably 
saved into the transactional data base. 

Fig. 36 is a diagrammatic representation of the AHP decision engine process. First, at 
3601 the data-grid saved in the transactional data base is opened and the format and field 

25 types are validated At 3602, the data-grid is loaded into a table specified for evaluation by 
the AHP evaluation engine. Next, at 3603 utility curves, rating scales, step scales, and direct 
weights are applied to cells in the data-grid. See AHP evaluation sub-process (shown in Fig. 
39). At 3604, the profile and mapped hierarchy saved in the profile database is opened and 
the preference weights are calculated using the AHP evaluation sub-process (shown in Fig. 

30 40). Next, at 3605 a table is created that contains alternatives with their AHP-assigned 



45 



WO 01/20530 



PCT/US00/25506 



rankings and corresponding scores and at 3606 the table is saved, with the ranked 
alternatives, along with the associated data, into the transactional database. 

Figs. 36A-C show examples of the steps of 3603 and 3604 in more detail. In Fig. 
36A, the covering objectives or characteristics are shown in a row along the top of the table 
5 and the alternatives, corresponding to for example various sellers' products, are shown in a 
column along the left hand side of the table. In Figs. 36A and 36B, various types of utility 
curves are shown in the second row of the table for display purposes only to demonstrate the 
types of curves that may correspond to the objective of that column. In Fig. 36A, the table is 
populated by various raw scores for each alternative that corresponds to each objective. In 

1 0 Fig. 36B, each raw score is transformed according to the utility curve associated with the 
correlated objective of that column into an alternate value score. In Fig. 36C, the global 
priority of each objective is determined by the AHP process (discussed above) and multiplied 
by each alternate value score for each objective and the summation of these multiplied values 
for each alternative yields a total score for each alternative. For illustration purposes only, 

15 global priority numbers have been inserted for the various covering objectives. As previously 
described, the sum of the global priorities for all of the objectives equals one. 

Fig. 37 shows a preferred graphical sensitivity output method for use in this 
embodiment so that the user can dynamically alter the user preferences and simultaneously 
view the results. First, at 3701 the table containing ranked alternatives is opened. This table 

20 is preferably displayed to the user and reflects the results of the evaluation. The table is 

preferably populated with information from the AHP-ranked table discussed previously. The 
layout may be custom designed for each application. Next, at 3702 the user may choose the 
alternatives to include in the final evaluation by selecting them. At 3703-3707, the user can 
select a dynamic graph tool to evaluate the results based upon the chosen alternatives. As 

25 previously discussed, the user can choose from a dynamic sensitivity graph tool at 3704, a 
dynamic gap graph tool at 3705, a head-to-head tool at 3706, or a custom output tool at 3707 
that can be designed specifically for each application. At 3708, the weights can be altered or 
updated and the results can be saved in the transactional database. The user may save, 
update, or rework the profile as desired. Upon a save request, the profile database is also 

30 updated. 
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Sensitivity analysis of a subset of alternatives may also be performed by the user. 
Selections of those alternatives are preferably fed into a sub-process dynamic sensitivity 
(shown in Fig. 41). Next, the subset of alternatives selected by the user are preferably fed 
into the sub-process dynamic performance (shown in Fig. 42) and then into the sub-process 
5 head to head (shown in Fig. 43). The custom outputs can be delivered and populated with . 
data from the alternative subset. 

Referring to Figs. 38-42, process sub-module diagrams are shown. Fig. 38 shows a 
preferred dynamic tree building module which is a part of the method shown in Fig. 32. First 
a form is delivered to the user's local system for the user to complete. Preferably, the module 

1 0 is developed by a content expert and the content of the form has mapped links to the 
hierarchy. Also preferably, the form is configured as an applet written in the Java 
programming language; however, other configurations may be used. The applet is preferably 
in communication with a hierarchy database and a dynamically-generated data table. The 
mapping and business rules are contained in the hierarchy database and data used for 

1 5 validation is housed in the dynamically-generated data table. The module preferably captures 
the user's responses the form. The business rules are applied to validate that the user 
responses have resulted and that both adequate filter questions and a navigable hierarchy with 
appropriate data points is linked to them. 

Referring to Fig. 39, a preferred trade-offs module is shown for this embodiment. 

20 Preferably the module is configured and performed in the form of a Java applet delivered to 
the user's local system, allowing the user to perform trade-offs or pairwise comparisons of 
any node of the hierarchical tree. The module preferably contains modified programming 
code that loads objectives into the trade-offs module navigation tree and calculates 
inconsistency, as described above. Also, the calculated inconsistency values can be compared 

25 to a pool of demographically-similar users so as to provide a more accurate comparison upon 
which the user can decide whether or not to proceed with certain inconsistency values and to 
ensure that the inconsistency is within the bounds of validity. Inconsistency thresholds can be 
set based upon the components of each unique decision hierarchy. Also, business rules are 
applied to validate that the user responses have completed the necessary number of trade-offs 

30 or comparisons, and if not the user is given the choice to either load a profile or continue to 
perform trade-offs or pairwise comparisons. After the trade-offs are complete, the decision 
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engine running AHP is applied to derive and assign preference weights, as previously 
described. The user is then given a choice, for example via a standard html web page, to save 
his/her preferences or save them as a profile to be used later and applied to similar decisions 
in the future. 

5 Referring to Fig. 40, a preferred utility curve application is shown. The application is 

delivered to a user's local system and allows the user to administer the utility curves. The 
user can set and apply utility curves or ratings scales to adjust the application of the curves. 
For example, the user can set minimum and maximum values, slope, or ranges for step scale. 
The minimum and maximum values can be used to update the utility curves based upon the 
1 0 intervals chosen to create a final ranking for the interval applied to the product data supplied 
by the net market. In this way, a more focused evaluation can be performed, depending upon 
the criteria and utility curve that is applied. Also preferably, the utility curve application can 
be self-adjusted for a set of data points so that a user does not have to administer to the utility 
curve. 

1 5 Furthermore, if the user is an expert, the utility curve application can allow the expert 

to build utility curves or ratings scales or step functions. Using this application, an expert 
user, such as an engineer, can create a utility curve by rating a product or characteristic based 
upon his/her expertise. As previously, described, the ratings assigned to the products are 
provided on the Y axis and the properties of the relevant characteristic on the X axis to create 

20 the utility curve. Such a utility curve enables the system to capture the expert's judgements 
and apply different scores to different properties of the product. The utility application is 
preferably written in the form of a java applet and delivered to the expert user's local system; 
however, any other suitable programming language may be used. The module is capable of 
reading in data and capturing data dynamically in a data table or aggregated file in the 

25 hierarchy database so that a utility curve can be dynamically created, altered or updated. In 
this way, a plurality of expert ratings can be combined, for example, using AHP to synthesize 
the judgments, to yield a utility curve reflective of numerous expert ratings. The user or 
participant can apply any scale and may assess a predefined ratings scale so as to have it . 
reflect his/her explicit judgments. An internal module of the application preferably reads the 

30 data input feeds and sets the min and max of continuous data utility curves to represent the 
actual data while holding the slope of the formula constant. 
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Referring to Fig. 41 , a diagram of a preferred evaluation ranking engine is shown. 
First, an applet, preferably written in the Java programming language, is delivered to the 
user, preferably via HTML so that the user may view the ranked alternatives and select a 
subset to evaluate with graphical tools. Ranking is done as previously described by 
performing a combined synthesis of the AHP generated preference weights and the converted 
and normalized data scores. Next, a user may select a sub-set of alternatives loaded into the 
transactional database to be used in graphical outputs to further evaluate the rankings. 

Referring to Figs. 42-44, views of the preferred graphical analysis tools used in the 
present invention are shown generally. These figures show each different graph as being 
linked to a dynamically generated data table. Figs. 42 and 43 illustrate a preferred method of 
adjustment of dynamic sensitivity graphs. First, an applet is preferably delivered to the local 
system of the user containing a dynamic , sensitivity graph or a dynamic gap analysis graph, as 
discussed above. A graph builder module preferably loads a sub-set of selected alternatives 
and allows the user to perform a dynamic sensitivity adjustment with respect to any node of 
the hierarchy. Also preferably, the user can simultaneously view components and the results, 
as well as revert and/or save. Preferably, a sub-module is programmed to allow the user to 
perform sensitivity adjustment to the selected sub-nodes as well. The user can save the 
adjusted profile as a separate profile or update other profiles. 

Referring to Fig. 44, a preferred method of performing static head-to-head graph 
analysis is shown. An applet is delivered to the local system of the user containing the head- 
tp-head graph as previously described. A graph builder module is preferably loaded with a 
subset of selected alternatives and allows the user to select which alternatives to view with 
respect to any node of the hierarchy. The user can view components, revert, or save. The 
user can save the adjusted profile as a separate profile or update previously-saved profiles. 
Also, a configurator application can be integrated into the evaluation module so that if a 
sufficient match is not obtained, the configurator application can construct a purchasing 
profile and select a product using the resulting profile so that the product can be customized 
to the profile. 

Figs. 45-55 provide samples of input and output screens for the second preferred 
embodiment shown, for example, implemented in the procurement of aircraft parts. Figs. 
46A, 49 A. 50A, 51 A, 54A, and 55 A depict screens analogous to those shown in Figs. 46, 49, 
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50, 51, 54, and 55, respectively, but for automobile procurement instead of aircraft parts 
procurement. In .Fig. 45, first a user, such as a purchasing manager from an aircraft 
manufacturer, can log on to a net market or web portal through a network. Next, as shown in 
Fig. 46, the user is presented with filtering questions as explained above. In Fig. 47 
5 horizontal slider bars are presented to perform pairwise comparisons on high level criteria. In 
Fig. 48 the user is reminded to perform lower level trade-offs and in Fig. 49 horizontal slider 
bars are presented to perform pairwise comparisons on sub-factors or lower level criteria. 

Fig. 50 is a screen view of a results page illustrating the product, here an aircraft 
engine, which best fits the user's wants and needs. Preferably, the results will link users to 
1 0 product specifications, product reviews, and procurement sites where a transaction can be 
consummated. 

Figs. 51-53 illustrate a dynamic sensitivity analysis of the relationship of the weights 
and the priorities of the resulting selections, e.g., the engines. In Fig. 5 1 , the rankings for the 
engines are shown on the right in the form of a horizontal bar graph, and the objectives or 

15 characteristics are indicated on the left in a group of individual horizontal slider bars. The 
user can dynamically change the illustrated horizontal bars corresponding to the 
characteristics so as to change their weights and cause the system to recompute the product 
rankings. In Fig. 52, the results are ranked and the horizontal bar graph corresponding to the 
ranked products is shown broken down into components. In this way the user can visually 

20 inspect how each factor contributed to the overall result Also, a user may decide that 

performance is a more important goal and the user may drag the performance bar to the right 
to indicate such a change of mind. For example, in Fig. 53 the product rankings have been 
altered as a result of the user changing the preference of performance from 7% in Fig. 52, to 
39% in Fig. 53 and, in this example, a different engine has a higher ranking. Fig. 54 shows 

25 an exemplary dynamic gap analysis graph similar to that of Fig. 15 to assist the user in 
determining how each product performs against the other with respect to each factor. 

Fig. 55 provides a static comparison of characteristics of two alternatives. This plot is 
known as head-to-head comparison. For example, the screen of Fig. 55 illustrates a 
comparison between the General Electric 1 158 and the Rolls Royce RB21 1-524, wherein 

30 price and maintenance are rated higher for the Rolls Royce and performance, delivery and 
fuel efficiency are rated higher for General Electric . Also, the overall score is higher for 
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General Electric. This display can be constructed based on the derived weights and ratings of 
the alternatives. 

To further describe the second preferred embodiment and related embodiments, it is 
useful at this point to compare the similarities and differences between the first and second 
5 preferred embodiments. 

Referring to Figure 56, in the first preferred embodiment a buyer (or user - they are 
synonymous in this description) accesses at step 5605 a system RFX (request for proposal 
(RFP) or request for quotation (RFQ)) or product evaluation engine by directing a browser to 
a previously established URL for a preferred system. The system responds by presenting the 
10 buyer with a login screen at step 5610 (see also Figure 68, step 6810), as is known in the art. 
The buyer is presented with three options: (1) enter the site anonymously; (2) enter an 
established user name; or (3) establish a new user name by typing one in and answering 
demographic questions that will be associated with all of the buyer's purchasing profiles. 
After successfully logging in to the site (site and portal are used interchangeably in 
15 this description) the buyer is required at step 5615 (see also Figure 68, step 6815) to select a 
model. A model selection interface is depicted in Figure 59. A model has a one-to-one 
relationship with a product, although a product may have a one-to-many relationship with a 
model. For example, a product may have multiple types of models that a buyer can use to 
evaluate offerings by multiple suppliers, but each model can only evaluate one pre-specified 
20 type, class, or category of product. A model is preferably composed of 5 essential elements: 
(1) objectives, organized in a parent-child structure that when transversed creates a decision 
tree; (2) a set of rating scales that have a one-to-one correspondence with the covering 
objectives of the decision tree; (3) a set of filter or needs questions related to the entire 
alternative or product database; (4) informative text to educate, guide, and clarify terms for 
25 the buyer; (5) sequence, rules, and flow-of-analysis applets and a tradeoffs applet. 

Next in the process, at step 5620 (see also Figure 68, step 6820), the buyer selects a 
decision profile (see Figure 60), each of which has a many-to-one relationship with a model. 
This structure is depicted in Figure 58: one model can have many profiles but a profile is 
specific to one and only one model. There are two types of decision profiles available to a 
0 buyer: local and global. A local decision profile is one that was created by or specifically for 
a buyer; it can be modified at the buyer's discretion. A global decision profile (the default 
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profile is global) is created by the operator of the system site. It is available to all users and 
may not be modified by any buyer. A buyer may, however, make changes to a global 
decision profile and save it under a new name as a local profile. It is important to note that a 
decision profile is related to a buyer's demographic information, but the buyer's decision 
5 profile is actually composed of 3 essential elements: (1) a set of responses to filter questions; 
(2) a set of judgments relating to fee model's tradeoffs; and (3) a selected set of rating scales. 
Demographic information can be related to many profiles owned by one buyer, but it is not 
"contained" in the profile. A buyer is not required to supply demographic information in 
order to maintain a decision profile. 

1 0 Once a profile is selected by the buyer and loaded by the system, the buyer has several 

choices available. The buyer may choose to perform all steps in the process and modify or 
verify the selected profile, or may visit either filters or tradeoffs independently in order to 
modify decision profile settings. Or the buyer may choose to go directly to results, bypassing 
both filters and tradeoffs; settings of the selected decision profile will be used to evaluate the 

1 5 resulting set of selected products or alternatives. These options are represented in FIG. 56 by 
the dotted lines. 

At step 5625 (see also Figure 68, step 6825), needs or filters are administered. During 
this step the buyer selects a set of business terms and product specifications (see Figure 61) 
which an alternative or product is required to meet in order to be included in the evaluation 
20 process. Examples of this may be a lead-time of less than 30 days, four ports, a footprint of 
less than 4 sq. ft, or 24/7 technical support. The results from the filter questions are 
transformed into standard SQL queries and used at step 5630 (see also Figure 68, step 6830) 
to cull down the existing product or alternative database. The result of this process is a final 
set of unranked alternatives or products available to the buyer. 

25 Tradeoffs are the next step (step 5635) (see also Figure 68, step 6835) in the process. 

The buyer is presented with an applet that delivers all of the decision tree's objectives 
organized in pairs. See Figure 62. Corresponding to these pairs are bars that can be 
dynamically slid left or right in order to represent one of 3 relations between the objectives: 
(1) importance; (2) preference; or (3) likelihood. The buyer completes these comparisons by 

30 setting the bars to represent one of three previous conditions of the one-to-one relationship. 
See Figure 63. For example, perhaps safety is viewed as 20% more important than 
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performance when buying a car, or leather seats are 40% more preferable than a 5 disk CD 
player in a new car. The visual results of the tradeoffs are the ratios of the lengths of the top 
lines to the lengths of the bottom lines (see Figure 62). These ratios are called jvdgments, and 
are stored in arrays or matrices. Once these are populated, the system initiates AHP 
5 mathematics (see FIGS. 21-24 and accompanying text), which at step 5640 (see also Figure 
68, step 6840) calculates the buyer's priorities for the purchasing objectives. 

With the alternative set identified, the next step (step 5645) (see also Figure 68, step 
6845) is to apply rating scales. The rating scales are preferably defined by the portal owner or 
customer. The definition can be accomplished, for example, with the assistance of 
1 0 ExpertChoice development software, available from ExpertCommerce, Inc., 6 East 32nd 
Street / 4th floor, New York, NY 1 001 6. Preferably, there is a rating scale assigned to each 
and every covering objective. Missing data for a covering objective is not critical, as the 
synthesis algorithms (see Figs. 24 and 36 and accompanying text) overcome this by not 
assigning a value score to that covering objective. This means that those alternatives or 
1 5 products that do not have complete information are only penalized the proportional and 
specific amount of the missing data point, as defined by the buyer. The rating scales are 
applied by taking the transformed raw score and normalizing it by running it through either a 
utility curve (see Figs. 19, 25, 26, 36, 40 and accompanying text), a rating scale, or an interval 
step scale. The result is a value score between 0 and 1 . 

20 Once the system has the data points' or covering objectives' value scores, at step 5650 

(see also Figure 68, step 6850) synthesis is performed to assign an overall value score and 
corresponding rank to the alternatives or products. Synthesis is the process that brings the 
buyer's priorities and the product's specifications together. See Figs. 24 and 36. The overall 
value score, like the data point value score, is a number between 0 and 1 . The alternatives are 

25 sorted based on the overall value scores, and displayed at step 5655 (see also Figure 68, step 
6855) to the user in a dynamic results page, depicted in Figure 64, that allows the user to re- 
sort the alternatives by any of the alternatives' factors. The results page contains links to 
analysis tools, tabular information on the actual specifications, and business terms, as well as 
links to initiate transactional actions. 

30 Another sub-process is available at this point in the process: feedback. Calculations 

are run across priorities and alternatives so that feedback (see Figure 27 and accompanying 
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